If you have devoted some time to review the new API, maybe you have discovered an odd thing: there are some functions missing. Ok, you can blame me for remove *that* function doing exactly what you need. Of course that may be my mistake. But please consider perhaps I have good reasons to do that.
Let's take one example:
cmsReadICCMatrixRGB2XYZ(LPMAT3 r, cmsHPROFILE hProfile);
This function no longer exists in lcms 2.0 API.
Well, many people were using this function to retrieve primaries of a profile. So, for example, if you want to know which are the AdobeRGB primaries, just call the function with the right profile and here you go.
Seems easy, and useful, but trust me, it is not. The real reason d'être of this function is somehow surprising. Not because it is handy but because is precise. Please consider this piece of pseudo-code:
cmsXYZTRIPLE Result;
hXYZ = cmsCreateXYZProfile()
xform = cmsCreateTransform(hProfile, TYPE_RGB_DBL, hXYZ, TYPE_XYZ_DBL, INTENT_RELATIVE_COLRIMETRIC, 0)
cmsDoTransform(xform, {{ 1, 0, 0}, {0, 1, 0}, {0,0,1}}, &Result, 3)
Do you follow it? I create a transform from the profile (in RGB) to XYZ. Then I convert max of R, and B to XYZ. I am obtaining the primaries! Despite it seems more complex, this method is much better because is guaranteed to work in *any* profile, not only on matrix-shaper ones.
So, what is the point of having the old function? Easy: lcms 1.x was precission-limited to 16 bits, so you cannot obtain primaries with enough precision with the method described above. But that does not apply with lcms2, where you have an outstanding 64-bit double precission. Less is more in this particular case!
Monday, July 27, 2009
Thursday, July 23, 2009
Linking tags
Implementing the tag link feature has been easier than I originally thought, but there are some caveats.
lcms2 does support some new features on read/writting profiles. You can use cmsReadTag/cmsWriteTag to read/write lcms objects like LUTs, tone curves and so. You can also use cmsReadRawTag/cmsWriteRawTag to read or write whatever you want, but poor library does not do any checking or understanding on whats going on.
You can also link tags to items created by any of those two methods. So far so good.
But wait, you can also write plug-ins to add more "understood" objects in cmsReadTag/cmsWriteTag and also you can write plug-ins to add new types for those objects.
In addition to all that, there is a brand new structure added as an addedum to ICC spec 4.2, that is the MPE or multi profile elements. This is worth of several comments in this blog. Basically that may make photographers and precission yonkies very happy as it includes *true* floating point numbers, among other things. There is a plug-in type devoted to MPEs. More to come.
So, if you consider all all those acess methods should be consistent, we have a nice mess. The good news are it seems to work, the bad news, we need more testing. But overall I see progress.
Fear not!, maybe I will even accomplish the schedule and realease the whole thing on November.
lcms2 does support some new features on read/writting profiles. You can use cmsReadTag/cmsWriteTag to read/write lcms objects like LUTs, tone curves and so. You can also use cmsReadRawTag/cmsWriteRawTag to read or write whatever you want, but poor library does not do any checking or understanding on whats going on.
You can also link tags to items created by any of those two methods. So far so good.
But wait, you can also write plug-ins to add more "understood" objects in cmsReadTag/cmsWriteTag and also you can write plug-ins to add new types for those objects.
In addition to all that, there is a brand new structure added as an addedum to ICC spec 4.2, that is the MPE or multi profile elements. This is worth of several comments in this blog. Basically that may make photographers and precission yonkies very happy as it includes *true* floating point numbers, among other things. There is a plug-in type devoted to MPEs. More to come.
So, if you consider all all those acess methods should be consistent, we have a nice mess. The good news are it seems to work, the bad news, we need more testing. But overall I see progress.
Fear not!, maybe I will even accomplish the schedule and realease the whole thing on November.
Wednesday, July 22, 2009
ZOO Doppelgänger and the Link feature
To some extent, old'n'good lcms allowed profile editing. That was a "dangerous" feature in the sense people may abuse of it to grab copyrighted material. That is, you may open your favourite profile to remove the copyright tag, and here you go. Obviously preventing that by the hard way is just like killing the messanger, so this feature is working in lcms and is up to you to use -or abuse- it.
But reviewing the previous entry about the zoo, I had a wild idea: if the zoo test reads every single tag, what if then I try to rewrite all those tags? that would create a Doppelgänger version of every profile in the zoo, but not necesarely with same organization and size.
The test code was written in few minutes, and after I run it, poor lcms2 get into hyperspace and crashed badly.
Ok, five bugs ahead, I got the writting/copying feature working. But this also has unveiled how profile vendors abuse of the link feature. That is, since a tag is described by some block in the file, I can put two or more different entries in the tag directory pointing to the same location. Humm.. I have to add some code to deal with this case.
But reviewing the previous entry about the zoo, I had a wild idea: if the zoo test reads every single tag, what if then I try to rewrite all those tags? that would create a Doppelgänger version of every profile in the zoo, but not necesarely with same organization and size.
The test code was written in few minutes, and after I run it, poor lcms2 get into hyperspace and crashed badly.
Ok, five bugs ahead, I got the writting/copying feature working. But this also has unveiled how profile vendors abuse of the link feature. That is, since a tag is described by some block in the file, I can put two or more different entries in the tag directory pointing to the same location. Humm.. I have to add some code to deal with this case.
Friday, July 17, 2009
Profile ZOO
The serialization part is now complete. That is, lcms2 should be able to read all tags it understands on all profiles in the wide world. Since it understands all tags that are, or have been part of any ICC spec, present or past, a lot of profiles should be readed by current code.
I have been lately worried about stability and qualification. If lcms2 want any success, it should be tolerant with ill-formed profiles. I say tolerant, not permissive, because crafted profiles may be used by the bad guys to introduce exploits. We don't want wargames again, right?
I have compiled along the +10 years of lcms life, a "Zoo" of profiles. This collection includes actually about 1500 assorted profiles, going from the widely distributed v2 sRGB to rare corner cases, like devicelinks holding 12-ink separations. Not to be very common.
Some of those profiles are broken. Well, not completely broken. They have slightly malformed tags, like colorants using a bad type, descriptions with wrong char count, bad sizes in the header...
To check how well lcms2 may deal with that, I wrote a small program that runs across every single profile in the Zoo and then reads every tag in the profile. The code should reject the unuseable tags and behave nicely if some information can be recovered. It cannot in anycase segfault or leak resources.
So, keeping my fingers crossed, I executed the program and ... almost! one segmentation fault. Ok, it was a bug, fixed. Run it again and .. success! no memory leaks, lots of tags discarded and a cool "all is ok" printf'd at the end. Pfew!
I have been lately worried about stability and qualification. If lcms2 want any success, it should be tolerant with ill-formed profiles. I say tolerant, not permissive, because crafted profiles may be used by the bad guys to introduce exploits. We don't want wargames again, right?
I have compiled along the +10 years of lcms life, a "Zoo" of profiles. This collection includes actually about 1500 assorted profiles, going from the widely distributed v2 sRGB to rare corner cases, like devicelinks holding 12-ink separations. Not to be very common.
Some of those profiles are broken. Well, not completely broken. They have slightly malformed tags, like colorants using a bad type, descriptions with wrong char count, bad sizes in the header...
To check how well lcms2 may deal with that, I wrote a small program that runs across every single profile in the Zoo and then reads every tag in the profile. The code should reject the unuseable tags and behave nicely if some information can be recovered. It cannot in anycase segfault or leak resources.
So, keeping my fingers crossed, I executed the program and ... almost! one segmentation fault. Ok, it was a bug, fixed. Run it again and .. success! no memory leaks, lots of tags discarded and a cool "all is ok" printf'd at the end. Pfew!
Tuesday, July 14, 2009
Same profile on both sides
It is very convenient to detect whatever the source and destination profiles are same to instruct the CMM to do nothing. Seems quite simple but it is certaily complex.
The issue is on embedded profiles. You can't do a binary compare because embedded profiles may have changed attributes. That is, some fields in the profile header are different to reflect the preference on intent and the fact the profile is being used embedded.
V4 offersProfileID, which is an MD5 checksum of the profile avoiding those conflicting fields. Which is a good thing: if both source and destination profiles does have same ProfileID AND the intent is same on both profiles, then you can get rid of the whole transform as it is basically a no-op.
But sometimes (most of times, currently) you get AdobeRGB or sRGB embedded, which are v2 profiles. No Profile ID, and a very common case.
So, let's try to do some optimization. If both profiles are matrix-shaper, you can detect if the obtained matrix is an identity, and then if the curves are cancelling. We have room for improvement in 3 cases:
The issue is on embedded profiles. You can't do a binary compare because embedded profiles may have changed attributes. That is, some fields in the profile header are different to reflect the preference on intent and the fact the profile is being used embedded.
V4 offersProfileID, which is an MD5 checksum of the profile avoiding those conflicting fields. Which is a good thing: if both source and destination profiles does have same ProfileID AND the intent is same on both profiles, then you can get rid of the whole transform as it is basically a no-op.
But sometimes (most of times, currently) you get AdobeRGB or sRGB embedded, which are v2 profiles. No Profile ID, and a very common case.
So, let's try to do some optimization. If both profiles are matrix-shaper, you can detect if the obtained matrix is an identity, and then if the curves are cancelling. We have room for improvement in 3 cases:
- All different
- Same primaries but different gamma
- Same primaries and equal tone curves
Saturday, July 11, 2009
More on speed
As promised, I have updated the snapshot. The performance numbers on matrix-shaper to matrix shaper should be close to what lcms2 is going to deliver when released. If you want to run the testbed, you would need to copy those profiles from Photoshop distribution, as I'm not allowed to redistribute that:
Then take a look on the numbers at the end of the testbed execution.
tifficc utility should also work to some extent, but there is a 1-pixel caché that may give bad performance. I have to turn caché off for such profiles as the caché code takes more than the transform itself.
It is funny to note that this is pure "C" code, and in some situations outperforms SSE2 hand-written assembly. That was the case when using the Intel compiler.
64-bits hardware is pretty untested, so if you manage to make it work on such architectures, please drop me a note, thanks!
- AdobeRGB1998.icc
- CoatedFOGRA27.icc
- UncoatedFOGRA29.icc
- USWebCoatedSWOP.icc
- USWebUncoated.icc
./configure; make; make check
Then take a look on the numbers at the end of the testbed execution.
tifficc utility should also work to some extent, but there is a 1-pixel caché that may give bad performance. I have to turn caché off for such profiles as the caché code takes more than the transform itself.
It is funny to note that this is pure "C" code, and in some situations outperforms SSE2 hand-written assembly. That was the case when using the Intel compiler.
64-bits hardware is pretty untested, so if you manage to make it work on such architectures, please drop me a note, thanks!
Thursday, July 9, 2009
about speed
I'm getting outstanding results with lcms2 and matrix-shaper profiles. It is still not on the public preview, so you have to trust me, but here are some numbers. That's tested on my laptop which is an old 2GHz 2-core CPU:
Thats a boost of about X 6.3. Please note that applies only to 8 bit matrix-shaper to matrix-shaper transforms, so RGB only! When primaries of both profiles are same, the performance is even better, it reaches about 30 Megapixel/second. I will put all this code available this weekend.
lcms 2.0:
8 bits on Matrix-Shaper profiles...done.
[625 tics, 0.625 sec, 25.6 Mpixels/sec.]
lcms 1.18:
lcms is transforming full spectrum in 8 bits...done.
[3984 tics, 3.984 sec, 4.01606 Mpixels/sec.]
Thats a boost of about X 6.3. Please note that applies only to 8 bit matrix-shaper to matrix-shaper transforms, so RGB only! When primaries of both profiles are same, the performance is even better, it reaches about 30 Megapixel/second. I will put all this code available this weekend.
Monday, July 6, 2009
Tag plug-in
Related with cmsReadTag, here comes one of the easier plug-ins to write: the tag plug-in.
Imagine you are a printer vendor and want to include in your profiles a private tag for storing the ink consumption. So, you register a private tag with ICC, and you get signature "inkc".
Ok, now you want to store this tag as a Lut16Type, so it will be driven by PCS and return one channel giving the relative ink consumption by color.
Writing a plugin in lcms2 will allow cmsReadTag and cmsWriteTag to deal with you new data exactly as any other standard tag.
To do so, you have to fill a cmsPluginTag structure to declare the plugin. This structure is formed by a base, which is common to all plug-ins.
plugin.base.Magic = cmsPluginMagicNumber;
plugin.base.ExpectedVersion = 2.0;
plugin.base.Type = cmsPluginTagSig;
That latter identifies your plug-in as "tag type". Now we need to define the tag signature
plugin.signature = 'inkc';
And some additional info about the type used by your tag:
plugin.descriptor. ElemCount = 1;
plugin.descriptor. nSupportedTypes = 1;
plugin.descriptor.SupportedTypes[0] = cmsSigLut16Type;
That is all. You can setup the new functionality by calling
cmsPlugin(&plugin);
Advanced tag plug-ins may use polymorphic types, depending on the version of the profile for example. Instead of one type, you can declare several. Then the read tag logic will search for all supported types to find the suitable one. cmsSigLut16Type for v2 and cmsSigLutBtoAType for v4 for example. There is an optional callback function to decide which type to use when writing the tag.
cmsTagTypeSignature DecideType(double ICCVersion, const void *Data);
This plugin is most useful when combined with the tag type plugin, which will be discussed soon.
Imagine you are a printer vendor and want to include in your profiles a private tag for storing the ink consumption. So, you register a private tag with ICC, and you get signature "inkc".
Ok, now you want to store this tag as a Lut16Type, so it will be driven by PCS and return one channel giving the relative ink consumption by color.
Writing a plugin in lcms2 will allow cmsReadTag and cmsWriteTag to deal with you new data exactly as any other standard tag.
To do so, you have to fill a cmsPluginTag structure to declare the plugin. This structure is formed by a base, which is common to all plug-ins.
plugin.base.Magic = cmsPluginMagicNumber;
plugin.base.ExpectedVersion = 2.0;
plugin.base.Type = cmsPluginTagSig;
That latter identifies your plug-in as "tag type". Now we need to define the tag signature
plugin.signature = 'inkc';
And some additional info about the type used by your tag:
- How many instances of the type the tag is going to hold (usually one)
- in how many different types the tag may come (again, usually one)
- and then the needed type(s).
plugin.descriptor. ElemCount = 1;
plugin.descriptor. nSupportedTypes = 1;
plugin.descriptor.SupportedTypes[0] = cmsSigLut16Type;
That is all. You can setup the new functionality by calling
cmsPlugin(&plugin);
Advanced tag plug-ins may use polymorphic types, depending on the version of the profile for example. Instead of one type, you can declare several. Then the read tag logic will search for all supported types to find the suitable one. cmsSigLut16Type for v2 and cmsSigLutBtoAType for v4 for example. There is an optional callback function to decide which type to use when writing the tag.
cmsTagTypeSignature DecideType(double ICCVersion, const void *Data);
This plugin is most useful when combined with the tag type plugin, which will be discussed soon.
Saturday, July 4, 2009
cmsReadRawTag
Today I've finished the cmsReadRawTag/cmsWriteRawTag interface. It may seem a poor accomplishment, but in reality that means the serialization engine is now complete.
In lcms2 you can read a tag from an open profile by doing
tag = cmsReadTag(hProfile, TagSignature)
And lcms will return (if found) a pointer to a structure holding the tag. Simple, but not simpler as the structure is not the contents of the tag, but the result of parsing the tag. For example, reading a cmsSigAToB0 tag results as a LUT structure ready to be used by all the cmsLUT functions. The memory belongs to the profile and is set free on closing the profile. In this way there are no memory duplicates and you can safely re-use the same tag. Writing tags is almost same, you just specify a pointer to structure and the tag name and lcms2 does all serialization for you. Process under the hood may be very complex, if you realize v2 and v4 of the ICC spec are using different representations of structures.
Anyway, you may decide all that is useless and your want just to write/read bytes to the profile, in this case the Raw variants are for you.
In lcms2 you can read a tag from an open profile by doing
tag = cmsReadTag(hProfile, TagSignature)
And lcms will return (if found) a pointer to a structure holding the tag. Simple, but not simpler as the structure is not the contents of the tag, but the result of parsing the tag. For example, reading a cmsSigAToB0 tag results as a LUT structure ready to be used by all the cmsLUT functions. The memory belongs to the profile and is set free on closing the profile. In this way there are no memory duplicates and you can safely re-use the same tag. Writing tags is almost same, you just specify a pointer to structure and the tag name and lcms2 does all serialization for you. Process under the hood may be very complex, if you realize v2 and v4 of the ICC spec are using different representations of structures.
Anyway, you may decide all that is useless and your want just to write/read bytes to the profile, in this case the Raw variants are for you.
Friday, July 3, 2009
Plug-ins
One of the main improvements of lcms2 is the plug-in architecture. Plug-ins means you can use the normal API to access customized functionality. Licensing are another compelling reason, you can move all your IP into a proprietary plug-in and still be able to upgrade core revisions in open source. There are 10 types of plug-ins currently supported.
cmsBool cmsPlugin(void* Plugin);
And the "Plugin" parameter may represent one or several plug-ins, as defined by the plug-in developer. To write plug-ins, there is an additional include file lcms2_plugin.h, which declares functions which are not in the public API but may be useful to this task. For example I/O access, matrix manipulation, and all the types needed to populate the plug-in structures. Those functions begins with "_cms" to denote those are extended functionality and should not be called the application by rather by the plug-in.
- Memory management
- Error management
- Interpolation
- Tone curve types
- Formatters
- Tag types
- Tags
- Rendering intents
- Multi processing elements
- Optimizations
cmsBool cmsPlugin(void* Plugin);
And the "Plugin" parameter may represent one or several plug-ins, as defined by the plug-in developer. To write plug-ins, there is an additional include file lcms2_plugin.h, which declares functions which are not in the public API but may be useful to this task. For example I/O access, matrix manipulation, and all the types needed to populate the plug-in structures. Those functions begins with "_cms" to denote those are extended functionality and should not be called the application by rather by the plug-in.
Thursday, July 2, 2009
The probe profile
"Jimmy Volatile" a Lcms user, suggested to incorporate test plots for the regression tests. In this way external apps using lcms could check if all is working as expected. I think this is a good idea, and maybe it is also feasible (all depends on the schedule).
An interesting check would be to use the ICC probe profile. This comes from the ICC site:
"The 'probe profile' (Probev1_ICCv2.icc) is syntactically a v2 ICC output device ('prtr') profile, and can be used in a workflow wherever such a profile is required. The color space of this profile is CMYK, and its PCS is Lab.
An interesting check would be to use the ICC probe profile. This comes from the ICC site:
"The 'probe profile' (Probev1_ICCv2.icc) is syntactically a v2 ICC output device ('prtr') profile, and can be used in a workflow wherever such a profile is required. The color space of this profile is CMYK, and its PCS is Lab.
Colors processed via this profile are deliberately distorted in a systematic way, to enable visual determination of the rendering intent used when rendering ("BToA" or PCS to device transforms) and when proofing ("AToB" or device to PCS transforms). This is useful, in cases when color-management-aware software does not document the behavior."
Subscribe to:
Posts (Atom)