Oncaphillis::niftyLib

Using the niftyLib::IO::device_streambuf

  1. First Encounter with the IOStream
  2. The niftyLib::IO::basic_streambuf
  3. The pluggable device_streambuf
  4. Making device_streambuf imbueable
  5. Concrete Examples: fd_device,streambuf_device,log_stream_device
  6. Filters

"First Encounter with the IOStream"

One of the most important parts of the Standard C++ Library beside the nifty containers and algorithms definitely is represented by the IOStreams system with its templated centerpieces std::basic_iostream and std::basic_iostreambuf and the full bouquet of their subclasses accompanied by a large set of manipulators and operators.

If you're coming from C, a transition like this:

// The old way
int main(int argc,char **argv) {
  printf(“Hello world, here comes %s”,argv[0]);
}

// The new way
int main(int argc, char **argv) {
  std::cout << “Hello world, here comes ” 
            << argv[0]
            << std::endl;
}

is probably your first encounter with IOstreams and with the whole C++-style. Instead of handing over arguments to a function which spits them out towards the users terminal you're applying an binary operator on arguments.

That is sooo cool and elegant just as if you're comparing the old style communicator of James T. Kirk with Picards small button on the chest and all software theorists even tell you that it is type safe since the ugly printf notation relies on the silent contract that arguments within the string (i.e. the s in our case) correspond to a fitting constconst char * in the arg list. So it's geeky and save at the same time.

There is a couple of ready made IOstreams for you to be used which all serve as a sink or source for C++ PODs and objects from the STL. Furthermore it's quite easy to roll your own input or output operator like:

std::ostream operator<<(std::ostream &os, const Complex & c) {
  os << c.real << “,” << c.imaginary;
  return os;

}

So with this you might just live for a long time or even happily ever after, but it becomes a little bit more complicated if you're not happy with the sinks or sources the standard IOStreams offer.

I came across this when I had to communicate with a UNIX style file descriptor (aka int) in a C++ conform way. My first naive approach was based on the assumption that the std::fstream should provide a CTor taking an int argument. The next idea was the sub classing of std::fstream cause thats what you first learn in OO programming. If your animal doesn't bark when you ask it to talk you just subclass and make it bark in the animal::talk virtual method.

So why not make your std::iostream tunnel the output to/from a different sink/source via overwriting a method called std::iostream::(write|read). Simply because their isn't such a method and input/output may be applied to the stream via a huge number of methods and operators. So it would take much more then a simple overwrite to make your animal behave like a dog.

The name IOstream actually may be a little bit misleading since the real work of streaming the formated characters to/from the terminal device is handed over to IOstreams close partner std::basic_streambuf<CharT,...>. iostream is merrily a input/output formatter which is responsible for converting your types into a chunk of characters while the streambuf (which doesn't have to do any buffering at all) takes care about kicking the device to do it's job.

It's std::basic_streambuf<CharT,...> which is the place to go if you'd like to make more devices available for the C++ IOStream system. It offers a clear set of virtual member functions which in spite of their frugal design offers a high degree of flexibility including such things as (a) unbuffered communication, (b) the possibility to seek, (c) code conversion based on the current locale etc.

In fact although it represents a very lean design it was worth to write a couple of large volume books about the art of IOStream handling and extending of which I might recommend this book of Langer and Kreft.

So in general you should write a new std::basic:streambuf<CharT,...> subclass whenever it comes to interfacing with a new device. However there is a whole class of devices which are quite happy with only two I/O calls for reading and writing, so I thought it might be worthwhile to further decompose the problem and come up with a pluggable device_streambuf concept.

The niftyLib::IO::basic_streambuf

The signature of the templated std::basic_streambuf class looks like:

temlate<class CharT,class TraitsT=std::char_traits<CharT> >
class basic_streambuf {
  ...
}

So a concrete class mainly has to be informed about the type of characters to be transported to/from the device. In a the pure and perfect world of template meta programming CharT might be just any type, but in reality only char or wchar_t will work properly. All the details about device handling and buffer management is hidden within std::basic_streambuf but one might easily modify the standard behavior (which actually refuses to read or write anything) by overwriting a couple of virtual methods namely:

std::basic_streambuf<CharT,...>::overflow

std::basic_streambuf<CharT,...>::bpackfail

std::basic_streambuf<CharT,...>::sync

std::basic_streambuf<CharT,...>::uflow

std::basic_streambuf<CharT,...>::underflow

Furthermore all subclasses have to manage the start, end and current position within internal read/write buffers via calls to the methods

std::basic_stream::<CharT,...>::setg and std::basic_stream::<CharT,...>::setp.

Based on that it was quite easy to come up with own niftyLib::IO::basic_streambuf<CharT,...> class which implements all the buffer handling part and tunnels all the dirty work into two pure virtual methods niftyLib::IO::basic_streambuf<CharT,...>read and niftyLib::IO::basic_streambuf::write. The full signature of the major parts looks like that:

template<class CharT,
         class PolicyT = basic_streambuf_policy,
         class TraitsT = std::char_traits<CharT> >
 class basic_streambuf : public std::basic_streambuf<CharT,TraitsT> {
   ...
   ...
   std::streamsize read(CharT  *,std::streamsize) 
               throw(Exception)= 0;
   std::streamsize write(CharT *,std::streamsize) 
               throw(Exception)= 0;
}

The PolicyT class mainly defines constants like buffers sizes and such, but may serve as a back door to further fine tune the basic_streambuf behavior.

The pure virtual methods are expected to return the number of successfully read or written characters on success, 0 when they hits eof and -1 on errors during reading and writing (e.g. trying to access a non existing device etc(. In case of real hefty errors (failure to allocate memories, device errors they also might throw exceptions of the niftyLib::IO::Exception type.

So now one might simply come up with h(is|er) own file based streambuf by simply subclassing and specializing as follows:

class my_fd_streambuf : 
  public niftyLib::IO::basic_streambuf<char> {
private:
  tyedef niftyLib::IO::basic_streambuf<char> super;
public:
  typedef typename super::char_type char_type;

  my_fd_streambuf(int i) : fd_(i) {
  }
  
  std::streamsize read(char_type * b,std::streamsize n) {
    return ::read(fd_b,n)
  }

  std::streamsize write(const char_type * b,std::streamsize n) {
    return ::write(fd_,b,n)
  }

private:
  int fd_
}

but you may push it even further by delegating the job of basic I/O to another class which can be passed in as a template argument.

The pluggable device_streambuf

That was the birth of the new subclass niftyLib::IO::device_streambuf with the following signature:

template< class CharT,
          class DeviceT = device<CharT,std::char_traits<CharT> >,
          class PolicyT = device_streambuf_policy<CharT,DeviceT>,
          class TraitsT = std::char_traits<CharT>,
          bool CodeCvt  = PolicyT::do_codecvt
        >  class device_streambuf {
};

So this one once again is supposed to transport characters of type CharT to some device which is given as another template arg and which defaults to niftyLib::IO::device<CharT,...>. The coupling between the niftyLib::IO::device_streambuf and the associated device however is loose. It is in no way expected to be of class (or subclass of) niftyLib::IO::device. It only has to provide the following typedefs and methods.

  // the character to be transported usually char or wchar_t
  // which defaults to CharT in device<CharT>
  char_type; 

  // traits_t define the type_traits for the given character
  // type. This defaults to std::char_traits<CharT>
  traits_type;

  // The method used to read from the buffer
  std::streamsize read(char_type *b,std::streamsize n)

  // The method used to write to the buffer

  std::streamsize write(constchar_type *b,std::streamsize n)

The niftyLib::IO::device<CharT> serves as a blueprint from which one might choose to subclass to get the desired default behavior. In that case only the device<CharT>::read and device<CharT>::write methods have to be overwritten. If however your device is supposed to be placed within a different class hierarchy you might as well choose any other class that provides the given types and methods.

The next template parameter PolicyT first of all serves the same purpose as for the niftyLib::IO::basic_streambuf. It defines constants which modify the inner behavior of the buffering process. In Addition to this it holds the standard value for the last template argument CodeCvt which plays an important role in turning the device_streambuf into an imbue-able streambuf. A property which will be discussed in the next chapter.

For now you might already do things like:

#include <iostream>
#include <niftylib/io/device_streambuf.h>
#include <niftylib/io/syslog_device.h>

using namespace niftyLib;

int main(int argc, const char ** argv ) {

  // Our own device_streambuf expecting to pump char
  // into a IO::device<char>

  IO::device_streambuf< char > * db = NULL; 

  try {

    // We allocate a new device_streambuf on the heap and give it 
    // a pointer to a syslog_device which is a subclass of
    // device<char> writing to
    // the systems syslog machinery. Since we didn't provide a second
    // arg to device_streams CTor it takes ownership of the 
    // associated device and we do not have to take care about
    // syslog_devices destruction

    std::ostream os ( db = 
      new IO::device_streambuf<char>( 
        new IO::syslog_device<char>() ) );

    // What else: Dump this out to the syslog
    os << "HELLO WORLD" << std::endl;
  } catch( std::exception & x) {
    // Since all nifyLib Exceptions are subclasses of std::exception
    // this should just work fine
    if(db!=NULL)
      delete db;
    std::cerr << "exception:what == '" << x.what() << "'" 
              << std::endl;
  }
  // We have to take care of out device_streambuf since 
  // std::ostream does not take ownership.
  delete db;
}

Making device_streambuf imbueable

So back to the last template argument of niftyLib::IO::device_streambuf. The bool CodeCvt determines if a concrete device_streambuf class should be imbueable. To make a stream or streambuf imbueabe means that it has to be aware of std::locale characteristics like number conventions and other stuff. So in general to call

std::cout.imbue(std::locale(“de_DE.ISO8859-15”))

would tell the standard output stream to use the german number and currency conventions beside many other stuff. It also tells the associated streambuf to use the character code page IO8859-15 (the ISO8859-1 variant including €). When dealing with type char for internal as well as for external character coding this mostly doesn't make any sense but it becomes important when working with wchar_t internally and feeding external devices with a stream of char. Therefor I designed the device_streambuf in a way that makes it perfectly legal to instantiate it as

...
  niftyLib::IO::device_streambuf<
             wchar_t,
             nifyLib::IO::device<char>,...>

So now we have a device_streambuf that is supposed to handle wchar_t internally but has to interface with a niftyLib::IO::device<char> which accepts char as it's input/output format. There definitely has to be some code conversion going on. This is indicated via the CodeCvt template argument which defaults to true whenever the internal (wchar_t in our case) type differs from the external type.

If it's false there will be no code conversion and nifytLib::IO::device_streambuf inherits directly from niftyLib::IO::basic_streambuf which as we have learned already more or less directly pushes/and pulls characters from and to the device.

If it CodeCvt happens to be true however niftyLib::IO::device_streambuf inherits from another std::basic_streambuf incarnation called niftyLib::IO::codecvt_streambuf in which the imbue method is not just a stub but forces a new std::codecvt facet on the internal buffers.

Its signature is:

template<class iCharT,
         class xCharT=iCharT,
         class PolicyT=codecvt_streambuf_policy,
         class iTraitsT=std::char_traits<iCharT>,
         class xTraitsT=std::char_traits<xCharT> >
class codecvt_streambuf : 
  public basic_streambuf< iCharT, PolicyT, iTraitsT >

So we deal with two different character types for internal and external character representation. We also inherit from basic_streambuf<iCharT,...>. codecvt_streambuf overwrites the pure virtual methods read and write as expected by any subclass of niftyLib::IO::basic_streambuf. In addition it offers a real imbue method which mainly ensures that the characters in the current input/output buffer are properly transformed to their internal/external representation.

So our previous example may be rewritten as:

#include <iostream>
#include <niftylib/io/device_streambuf.h>
#include <niftylib/io/syslog_device.h>

using namespace niftyLib;

int main(int argc, const char ** argv ) {

  // Now our internal characters are represented as wchar_t
  // externally we deal with char PolicyT guarantees that
  // the last arg CodeCvt defaults to true

  IO::device_streambuf<wchar_t,IO::device<char> > * db = NULL; 

  try {
    // Build a wostream communicating with our
    // device_streambuf
    std::wostream os ( db = 
      new IO::device_streambuf< wchar_t, IO::device<char> >( 
        new IO::syslog_device<char>() ) );

    // We have to place the proper code conversion here

    os.imbue(std::locale("ja_JP.utf-8"));

    // ... and of course have to deal with L"strings"
    // This should write a proper utf-8 stream into syslog

    os << L"HELLO 世界" << std::endl;

  } catch( std::exception & x) {

    // Since all nifyLib Exceptions are subclasses of std::exception
    // this should just work fine

    if(db!=NULL)
      delete db;

    std::cerr << "exception:what == '" << x.what() << "'" 
              << std::endl;
  }

  // We have to take care of out device_streambuf since 
  // std::ostream does not take care about that.

  delete db;
}

Concrete Examples: fd_device, streambuf_device, log_stream_device

Concrete examples for devices currently are:

fd_device<CharT>: which accepts a UNIX style file descriptor for its CTors arg. It reads/writes CharT from the given file descriptor which has to refer to an already open valid file.

..
int fd = ::open(“~/bla”);

std::iostream ios(
  new IO::device_streambuf<char>(
    new IO::fd_device(fd));

os << “hello world” << std::endl;

std::string s;

os >> s;

streambuf_device<CharT>:

uses another std::basic_streambuf<CharT> as its sink/source. e.g. The next code uses the streambuf of the std::stringstream for in place code conversion of a wchar_t data into a std::string encoded in utf-8.

#include <iostream>
#include <sstream>
#include <niftylib/io/device_streambuf.h>
#include <niftylib/io/device_streambuf.h>
#include <niftylib/io/streambuf_device.h>

using namespace niftyLib;

int main(int argc, const char ** argv ) {

  // Use the rdbuf of std::strinsgtream as a output device
  // externally we deal with char in utf-8 coding

  std::stringstream ss;
  IO::device_streambuf< wchar_t, IO::device<char> > * db = NULL; 

  try {
    // Create a wostream piping it's stuff into
    // our device_streambuf<wchar_t,IO::device<char>,...>
    std::wostream os ( db = 
      new IO::device_streambuf< wchar_t, IO::device<char> >( 
        new IO::streambuf_device<char>( ss.rdbuf() ) ) );

    // We have to place the proper code conversion here

    os.imbue(std::locale("ja_JP.utf-8"));

    os << L"HELLO 世界" << std::endl;

    // Dump out the utf-8 string. This only makes sense if the
    // current locale is a utf-8 derivate

    std::cerr << "[[" << ss.str() << "]]" << std::endl;

  } catch( std::exception & x) {

    // Since all nifyLib Exceptions are subclasses 
    // of std::exception
    // this should just work fine

    if(db!=NULL)
      delete db;

    std::cerr << "exception:what == '" << x.what() << "'" 
              << std::endl;
  }

  // We have to take care of out device_streambuf 
  // since std::ostream does
  // not take care about that.

  delete db;
}

syslog_device<CharT>: Writes into the syslog system of the operating system. Of course only supports writing.

Filters

Filters represent a special class of device<Char> which accept another device<CharT> as it's input/output sink/source. Currently only output filters are defined.

log_filter:

Serves as a virtual base class for everything we need for logfile writing.

Screens the output for newline characters and calls a virtual method whenever it encounters one. The standard method does nothing. Subclasses are welcome to insert a sequence of new characters into the current stream via the protected virtual method log_filte<CharT>::raw_write(const CharT *,std::streamsize n)This way it is easy to have a log file where every line starts with a given character sequence.Like:

// Subclass from IO::log_filter
// and overwrite line_intro so that
// “TEST -- “ gets written whenever
// we hit new-line

class my_log_filter : public niftyLib::IO::log_filter<char> {
   ...
   ...
   virtual std::streamsize line_intro() {
       return this->raw_write(" TEST -- ");
}

int main(int argc,char **argv) {

   std::device_streambuf<char> *d;
   // We use the streambuf of std::cerr as
   // the final destination for the data
   // 
   std::ostream os(d = new device_streambuf(
     new my_log_filter(
       new streambuf_device( std::cerr.rdbuf() ) ) );
         os << “Hello World” << std::endl
            << “Hello World” << std::endl;
}

This should print out:

TEST -- Hello World
TEST -- Hello World

str_log_filter:

Represents a subclass of the generic nifytLib::IO::log_filter which can be used to dump out a given string whenever a new line is encountered.

Like:

using namespace niftyLib;

int main(int argc,char **argv) {
   IO::str_log_filter<char> slf;
   IO::device_streambuf<char> *d;
   // We use the streambuf of std::cerr as
   // the final destination for the data
   // 
   std::ostream os(d = 
     new device_streambuf<char>(
       slf=new str_log_filter<char>(
         new streambuf_device( std::cerr.rdbuf() ) ) );

      // define the string to be written whenever
      // IO::str_log_filter sees a new-line
      slf->set_intro(“TEST#2 --”);

      os << “Hello World” << std::endl
         << “Hello World” << std::endl;
}

This should print out:

TEST#2 -- Hello World
TEST#2 -- Hello World

Note: Since str_log_filter works on the external character encoding it is more preferable to manipulate the current intro_string via the niftyLib::IO::log_intro stream manipulator which always accepts strings in the internal character encoding. So it is possible to do things like:

using namespace niftyLib;

int main(int argc,char **argv) {
   // Create a device_streambuf dealing with
   // wchar_t internaly writing to
   // a std::basic_streambuf<char> stolen
   // from std::cerr
   IO::str_log_filter<char> slf;
   IO::device_streambuf<wchar_t,IO::device<char> > *d;

   std::ostream os(d = 
    new IO::device_streambuf<wchar_t,device<char> >(
     slf=new IO::str_log_filter<char>(
      new IO::streambuf_device( std::cerr.rdbuf() ) ) );

   // log_intro stream manipulator
   // takes care about transforming the new
   // intro string into utf-8 representation
   // needed by the associated str_log_filter<char>
 
...os << IO::log_intro(L”あぶない”)

   os << L“ 世界" << std::endl;
}

This should print out:

危ない Hello 世界
危ない Hello 世界

time_log_filter:

Another subclass of IO::log_filter which dumps out the current time stamp at the beginning of each line. Also very nifty for log file writing.


Generated by  doxygen
© 2008; Dr. Sebastian Kloska ( Oncaphillis )
Powered by: [?]