The second challenge was uploading this huge amount of content with its metadata into Alfresco without huge hassle and in a reasonable timespan.
The Bulk Import tool
The tool we chose to import the content and data in Alfresco efficiently was the existing Bulk Import tool for Alfresco, developed by Peter Monks and available here: https://github.com/pmonks/alfresco-bulk-import.
The essence of how the tool works is that it recursively parses a folder structure on the disk and creates the file and folder structure accordingly in Alfresco. For each node to be created (folder or file), the Bulk Import tool is expecting an xml file containing the metadata for the node. Folders will be imported even without such a file associated, but all content needs to have a metadata xml present at the same location. The metadata xml can contain the type of the content, aspects and any standard or custom metadata you need to add to the file.
The beauty of open source comes into play here. We needed the Bulk Import tool to behave differently than standard - no problem, with the code available, we can customize the behavior to serve our needs.
We opted therefore to customize the Bulk Import tool, by making it use two additional parameters in the metadata xml files. These parameters specify the full path on disk where the files can be found and the full path in Alfresco where the files should be placed.
This way, we don’t need to create the folder structure locally as needed in Alfresco, since this operation would be too costly when it comes to millions of documents, no matter the method utilized. Also, we do not need to make any modifications to the original file locations at client side.
For the value of the target path in Alfresco, we needed two modifications:
- A change in the importItem method in the BatchImporterImpl class, to use the target path from the metadata xml instead of the default path for the upload, if the “rootNodeRef” parameter is present in the file:
- A new method in the same class (BatchImporterImpl) to create the path in Alfresco as specified in the “rootNodeRef” parameter, needed for the cases where the path does not already exist:
Performance wise, we were very happy with the choice of this multi-threaded bulk import tool. With two CPU cores and 8 gB RAM, the tool can upload around 25.000 documents per hour (documents of on average 500 kB).
Besides the above changes, we noticed a couple of other generic modifications that we needed to make to the standard Bulk Import tool to suit our needs and the needs of the client:
- If a file could not be found on disk, the whole import process got dropped due to a RuntimeException being thrown. This was far less than ideal because with such large amounts of content to import, you want to be able to allow the processes to run for hours without checking them regularly. For this reason, we modified the type of the exception thrown in case something goes wrong with the import, so that the node without content gets created, the error is logged and the process can continue. The change in the BatchImporterImpl class:
- We noticed that the cm:created and cm:name metadata values from the xml are not correctly set in Alfresco. Besides missing the correct information in the system, the wrong naming was also causing issues when running the import with “replace” option. We added checks in the code for these two parameters to ensure they are correctly set as per specified values in the metadata xml.
No universal solution for migrations
From our experience, each migration is different. The basis process is of course in essence the same, but tailoring the tools to the given conditions and limitations will always be necessary. In some situations, you need to extract data directly from the database, while in others you need to build it based on extracts in different formats (e.g. xml) from completely different systems.
For one client, we used Pentaho in combination with custom webscripts, for another we used Pentaho in combination with the standard Bulk Import tool and this time it was the customized Bulk Import tool in combination with a custom java batch file generator that did the job.
We will be happy to read about your experience or thoughts on content migrations around Alfresco or work together to find the smoothest and most efficient process for your unique scenario. Contact us!
By Miruna Chirita