Bug found while reviewing documents on how to create a custom metadata extractor. The original refactor had left the repo doing the mapping. It should have been passing the fully qualified repo properties to the T-Engine to do the mapping.
Linked to:
Alfresco/alfresco-community-repo#227Alfresco/acs-packaging#1826
ATS-829: Release T-Core (T-Engines) 2.3.6 [trigger release]
Linked to REPO-5219 Allow AGS AMP to specify metadata extract mapping
Added an extractMapping transform option to all metadata extractors to override the default one.
3rd party libraries to get a green build.
* Upgrade cxf-rt-transports-http and woodstox-core to avoid issues
* Upgrade to org.springframework.boot:spring-boot-starter-parent:2.3.5.RELEASE to avoid problem in org.springframework:spring-web
* Upgrade to activemq 5.15.13 to avoid problem in activemq-broker 5.15.12
* MNT-21869 libreoffice timeout set too high
Reduce default value of timeout for libreoffice from 2000min to 20min
Add option to configure libreoffice timeout externally.
Enable to configure externally the port on which the app starts.
Add external-engine-configuration.md
* ATS-762: Add Tika unit test for pdf to csv
* ATS-762: Fix indentation
* ATS-762: Added 3 tests for simple pipepline. msg > txt, txt > doc, txt > odt, txt > rtf
* ATS-762: Added tests for libreofficeToPdf pipeline
* ATS-762: Addressed Jan's comment about not using asterisk when importing modules
* ATS-762: Changed comment to pdf-->csv to address Jan's comment on the PR
* task/ATS-762_T: noticed the txt mime type was wrong so fixed it
Co-authored-by: kristian <kristian.dimitrov@alfresco.com>
* Metadata extract code added to T-Engines
* Required a refactor of duplicate code to avoid 3x more duplication:
- try catches used to return return exit codes
- calls to java libraries or commands to external processes
- building of transform options in controllers, adaptors
* integration tests based on current extracts performed in the repo
* included extract code for libreoffice, and embed code even though not used out of the box any more. There may well be custom extracts using them that move to T-Engines
* removal of unused imports
* minor autoOrient / allowEnlargement bug fixes that were not included in Paddington on the T-Engine side.
* ATS-731: Add half of the missing simple legacy transforms
* task/ATS-731_Update_T-Engines_config_with_remaining_legacy_transformers: Updated transforms list to include missing legacy transforms
* task/ATS-731_Update_T-Engines_config_with_remaining_legacy_transformers: Added missing Transforms
* task/ATS-731_Update_T-Engines_config_with_remaining_legacy_transformers: Added xml xltm, xlam, ppsx, ppsm, msg, and dita to pdf
* ATS-731: Remove depreciated workaround
* task/ATS-731_Update_T-Engines_config_with_remaining_legacy_transformers: Added more missing transforms. Only 6 left to do
* task/ATS-731_Update_T-Engines_config_with_remaining_legacy_transformers: Merged with Kristian's changes
* task/ATS-731_Update_T-Engines_config_with_remaining_legacy_transformers: Added Kristians' last commit back
* task/ATS-731_Update_T-Engines_config_with_remaining_legacy_transformers: Added final 6 missing transforms
* ATS-731: Remove unnecessary test configs (Tests now pull configs from jars)
* task/ATS-731_Update_T-Engines_config_with_remaining_legacy_transformers: Reverted libra office file back to how it was at Kristian's last commit
* task/ATS-731_Update_T: Took back out the outlook transforms so that they can be done by a pipeline inestead
* ATS-731: Read default engine configs from jars in tests
* ATS-731: Removed failing transforms from Image magick
* ATS-731: The branch now only contains the transforms that work and have been tested. Just one more needs to be added in libre office
* Added the last of the working transfroms
* ATS-731: Added one more transform
* Revert "ATS-731: Remove depreciated workaround"
This reverts commit 82de937e
* ATS-731: Enable info log level for the depreciated workaround
Co-authored-by: aharake <ayman.harake@alfresco.com>
* REPO-5188 SourceEncoding should not be used to select transforms as it is always provided to T-Engines
* Fix test. Now that SourceEncoding is in the options but is not used to select the transformer, we need to add another
option to force the use of the textToPdf transform rather than libreoffice.
Created test-jar for libreoffice-boot
Updated LibreOfficeControllerTest.java so it can provide inheritance
Updated LibreOfficeControllerTest#testPojoTransform to uses a viable targetMimetype
Fixed inconsitent naming for @Value annotation
Moved surefire config to super pom
Implement LibreOfficeIT on AIO Controller
Runs the tests inherited from the imagemagick-boot-test-jar through the All in One Transformer
Co-authored-by: Erik Knizat <erik.knizat@alfresco.com>
Co-authored-by: kristian <kristian.dimitrov@alfresco.com>
Co-authored-by: eknizat <26163420+eknizat@users.noreply.github.com>
* ATS-695/ATS-675 Add aio boot project
- Added the bare bones of a spring boot project to be used by aio. Currently based loosely on transform-misc.
* ATS-674/ATS-695 Add forms for each transformer.
* ATS-675/ATS-695 add empty test to pass build during dev
* ATS-695 remove maven profile to fix build
* ATS-675 Define interface and the aio transformer
* Fix formatting and rename the module as per review comments
* ATS-675/ATS-695 Add ProbeTestTransformation
Currenly uses MiscController implementation.
* ATS-675/ATS-695 Add logger method,
This will be code repeated in the local transform method and the processTransform method
* ATS-675/ATS-695 Implement local transform method
Minimum implementation for transform method.
* ATS-675/ATS-695 Implement processTransform
* ATS-675/ATS-695 Rename project to alfresco-transform-core-aio-boot
Add alfresco-transform-core-aio dependencies
* ATS-675/ATS-695 Fix build
Update project location
Update imports and variable declarations in TODOs
Add error handling.
Formatting.
* ATS-693: Update transform-misc Dockerfile with newly reserved uid
* Revert "ATS-691: Combine the win/linux pathToFile logic"
This reverts commit 61fe4820
* ATS-693: Update transform-misc Dockerfile with newly reserved uid
* "ATS-693: Add Dockerfile to aio-boot module"
* ATS-675/ATS-695 Add resource required for ProbeTestTrasform
* ATS-675/ATS-695 Remove test resources, to be added in test implementation
* ATS-693: Fix path to jar resources
* ATS-675/ATS-703 Moved Options builder to non boot jar.
* ATS-675/ATS-703 Rename OptionsBuilder to PdfRendererOptionsBuilder
This is to avoid confilct with OptionsBuilders in other T-engines.
* ATS-675/ATS-695 Added PdfRendererApadpter.java
Added dependency to pom.xml
Required transformation of String to Long, method added to Util.java
* ATS-675/ ATS-704
Implemented LibreOfficeAdapter
* ATS-675 Parity with base aio naming convention
* ATS-675/ATS-705 Implemented ImageMagickAdapter
Moved and renamed OptionsBuilder. Moved to alfresco-transform-imagemagick, renamed ImageMagickOptionsBuilder.
Added dependencies to pom.xml
* ATS-693: Implement maven docker build
* Initial tests
* Add initial tests for config aggregation
* Update AbstractTransformerControllerTest to use the new engine config names
* Fix up controller
* Fix travis tests (#205)
* Fix engine specific properties for engine config location
* Temporarily add engine configs to test resources for the boot modules. Will need to fix this properly
* Resolve some review comments
* ATS-675 - Move static strings to util class
* Refactor classes for simpler design (#210)
* ATS-702 Fix error handling
(cherry picked from commit e30cb5fda6ba2ae09c91ef61e69cba4689bcc8d9)
* ATS-675 Rename test class (fixes typo)
* ATS-675: Add aio transformer to static scan
* REPO-4639: Split tika engine_config.json into separate transformers.
* WIP: REPO-4639 Content conversion failed using Tika
The Tika T-Engine "transform" option does not exist when called via the Transform Service or Local transforms, which resulted in no transforms taking place. However this value is really not be needed as the T-Engine should be able to read its own engine_config.xml to work out which sub transform to use. Transforms only worked via Legacy transforms, which used a T-Engine.
This code is based on tried and tested ACS repository code. It has been further simplified.
TODO:
- replace the ConfigFileFinder class just added with something that uses Spring to read the JSON. i.e. simplify it.
- replace the CombinedConfig class just added with something that does not need the InLineTransformer. i.e. simplify it.
- create tests based on the repo tests
- remove the source and target mimetype checks in Tika as a check against engine_config.xml is cleaner.
- repeat the process for the Misc T-Engine as it has similar code checking source and target mimetypes.
- remove the transform option passed by the legacy transforms.
* Removed CombindConfig and ConfigFileFnder classes.
* Extracted AbstractTransformRegistry so that it may be used in the ACS repository too.
TODO AbstractTransformRegistry and AbstractTransformRegistry need to be moved to the alfresco-transform-model pakage
* tidy up only
* REPO-4639: Add priority to duplicate transforms.
* REPO-4639: Refactor TikaTransformationIT to use the new Tika /transform specifications
Changes AbstractTransformerControllerTest as the engine_config is now loaded in TransformRegistryImpl instead of AbstractTransformerController
* Rename to TransformServiceRegistry, so we don't have to change the repo code.
* Added the baseUrl parameter to the register method and fixed the missed rename in the last commit.
* Javadoc change only
* Moved common classes (with repo) AbbstractTransformRegistry and TransformServiceRegistry to alfresco-transform-model
* Replace (simplify) all the isTransformable calls with a check against the JSON.
- Tests now only pass targetEncoding to the 'string' transformer.
* Fix failing tests.
* Revert port change
* REPO-4639 : Add priorities to misc engine_config
* REPO-4639 : Add priorities to pdf-renderer and imagemagick engine_config
* Remove test that is @Ignored
* Pick up alfresco-transformer-model 1.0.2.7-REPO-4639-1
* REPO-4639 : Add priorities to libreoffice engine_config
* REPO-4639 : Add priorities to tika engine_config
* REPO-4639 : Remove all priorities with value equal to 50 (default) from engine_config
* Switch over to using TransformServiceRegistry in org.alfresco.transform.client.registry
Reintroduce the noExtensionSourceFilenameTest having removed @Ignore.
* New whitesource issue on commons-compress 1.18. Upgrading to 1.19.
* Removed the text/javascript -> text/plain test as this is not supported
* Modifications as a result of changes to method names in alfresco-transform-model
* Pick up alfresco-transform-model 1.0.2.7-ATS545-2
* Remove unused imports
- fix multiple Misc Transformer bugs related to file mimeTypes
- remove usage of 'source/targetMimetype' as transform options/parameters
- add 'source/targetMimetype' arguments to the 'processTransform' method
- remove unnecessary code (e.g. useless overridden methods)
- add quick* test resource files
- add integration test for 'Local Transformations' on the Misc engine
- set up Integration Tests POM configuration for all T-Engine modules
- move startup message from controllers to the Application classes (SpringBoot configuration beans)
- added static imports for most static variables and static methods
- simplified a few nested *if*s
- replaced Arrays.asList() with explicit immutable collections
- fixed a few IntelliJ code inspection warnings
- code refactoring:
if successful -> return the object ( Optional.ofNullable(request) )
if not successful -> throw an exception (instead of sending an error message on the queue)
- when calling 'convert()' method, catch this exception and send a T-Reply with error message on the queue
- add correlationID to error logs
