inteligr8/alfresco-transform-core
1
0
Fork 0
mirror of https://github.com/Alfresco/alfresco-transform-core.git synced 2025-08-14 17:58:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Save point: [skip ci]

Browse Source 
* example HelloTransformer
This commit is contained in:
alandavis
2022-07-20 17:34:16 +01:00
parent 6692946fb5
commit 79706f8510
22 changed files with 638 additions and 238 deletions
Download Patch File Download Diff File Expand all files Collapse all files

310
engines/base/README.md
View File

@@ -1,233 +1,147 @@
# Common code for Transform Engines
# Common base code for T-Engines
This project contains code that is common between all the ACS T-Engine transformers that run as Spring Boot process (optionally within their own
Docker containers). It performs common actions such as logging, throttling requests and handling the streaming of content to and from the container.
This project provides a common base for T-Engines and supersedes the
[original base](https://github.com/Alfresco/alfresco-transform-core/blob/master/deprecated/alfresco-transformer-base).
For more details on build a custom T-Engine, please refer to the current docs in ACS Packaging, including:
This project provides a base Spring Boot application (as a jar) to which transform
specific code may be added. It includes actions such as communication between
components and logging.
For more details on build a custom T-Engine and T-Config, please refer to the docs in ACS Packaging, including:
* [ATS Configuration](https://github.com/Alfresco/acs-packaging/blob/master/docs/custom-transforms-and-renditions.md#ats-configuration)
* [Creating a T-Engine](https://github.com/Alfresco/acs-packaging/blob/master/docs/creating-a-t-engine.md)
## Overview
A transformer project is expected to provide the following files:
A T-Engine project which extends this base is expected to provide the following:
~~~
src/main/resources/templates/test.html
src/main/java/org/alfresco/transformer/<TransformerName>Controller.java
src/main/java/org/alfresco/transformer/Application.java
~~~
* An implementation of the [TransformEngine](https://github.com/Alfresco/alfresco-transform-core/blob/master/engines/base/src/main/java/org/alfresco/transform/base/TransformEngine.java)
interface to describe the T-Engine.
* Implementations of the [CustomTransformer](engines/base/src/main/java/org/alfresco/transform/base/CustomTransformer.java)
interface with the actual transform code.
The `TransformEngine` and `CustomTransformer` implementations should have an
`@Component` annotation and be in or below the`org.alfresco.transform` package, so
that they will be discovered by the base T-Engine.
* test.html - A simple test page using [thymeleaf](http://www.thymeleaf.org) that gathers request
parameters, so they may be used to test the transformer.
The `TransformEngine.getTransformConfig()` method typically reads a `json` file.
The names in the config should match the names returned by the `CustomTransformer`
implementations.
~~~
<html xmlns:th="http://www.thymeleaf.org">
<body>
<div>
<h2>Test Transformation</h2>
<form method="POST" enctype="multipart/form-data" action="/transform">
<table>
<tr><td><div style="text-align:right">file *</div></td><td><input type="file" name="file" /></td></tr>
<tr><td><div style="text-align:right">file *</div></td><td><input type="file" name="file" /></td></tr>
<tr><td><div style="text-align:right">sourceExtension *</div></td><td><input type="text" name="sourceExtension" value="" /></td></tr>
<tr><td><div style="text-align:right">targetExtension *</div></td><td><input type="text" name="targetExtension" value="" /></td></tr>
<tr><td><div style="text-align:right">sourceMimetype *</div></td><td><input type="text" name="sourceMimetype" value="" /></td></tr>
<tr><td><div style="text-align:right">targetMimetype *</div></td><td><input type="text" name="targetMimetype" value="" /></td></tr>
<tr><td><div style="text-align:right">abc:width</div></td><td><input type="text" name="width" value="" /></td></tr>
<tr><td><div style="text-align:right">abc:height</div></td><td><input type="text" name="height" value="" /></td></tr>
<tr><td><div style="text-align:right">timeout</div></td><td><input type="text" name="timeout" value="" /></td></tr>
<tr><td></td><td><input type="submit" value="Transform" /></td></tr>
</table>
</form>
</div>
<div>
<a href="/log">Log</a>
</div>
</body>
</html>
~~~
* *TransformerName*Controller.java - A [Spring Boot](https://projects.spring.io/spring-boot/) Controller that
extends TransformController to handel requests. It implements a few methods including *transformImpl*
which is intended to perform the actual transform. Generally the transform is done in a sub class of
*JavaExecutor*, when a Java library is being used or *AbstractCommandExecutor*, when an external process is used.
Both are sub interfaces of *Transformer*.
**Example TransformEngine**
~~~
...
@Controller
public class TransformerNameController extends TransformController
The `TransformEngineName` is important if the config from multiple T-Engines is being
combined as they are sorted by name.
```
package org.alfresco.transform.example;
import com.google.common.collect.ImmutableMap;
import org.alfresco.transform.base.TransformEngine;
import org.alfresco.transform.base.probes.ProbeTransform;
import org.alfresco.transform.common.TransformConfigResourceReader;
import org.alfresco.transform.config.TransformConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
@Component
public class HelloTransformEngine implements TransformEngine
{
private static final Logger logger = LoggerFactory.getLogger(TransformerNameController.class);
@Autowired
private TransformConfigResourceReader transformConfigResourceReader;
TransformerNameExecutor executor;
@PostConstruct
private void init()
@Override
public String getTransformEngineName()
{
executor = new TransformerNameExecutor();
return "0200_hello";
}
@Override
public String getStartupMessage()
{
return "Startup "+getTransformEngineName()+"\nNo 3rd party licenses";
}
@Override
public TransformConfig getTransformConfig()
{
return transformConfigResourceReader.read("classpath:hello_engine_config.json");
}
@Override
public ProbeTransform getProbeTransform()
{
return new ProbeTransform("jane.txt", "text/plain", "text/plain",
ImmutableMap.of("sourceEncoding", "UTF-8", "language", "English"),
11, 10, 150, 1024, 1, 60 * 2);
}
}
```
**Example CustomTransformer**
```
package org.alfresco.transform.example;
import org.alfresco.transform.base.CustomTransformer;
import org.alfresco.transform.base.TransformManager;
import org.springframework.stereotype.Component;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Map;
@Component
public class HelloTransformer implements CustomTransformer
{
@Override
public String getTransformerName()
{
return "Transformer Name";
return "hello";
}
@Override
public String version()
public void transform(String sourceMimetype, InputStream inputStream, String targetMimetype,
OutputStream outputStream, Map<String, String> transformOptions, TransformManager transformManager)
throws Exception
{
return commandExecutor.version();
String name = new String(inputStream.readAllBytes(), transformOptions.get("sourceEncoding"));
String greeting = String.format(getGreeting(transformOptions.get("language")), name);
byte[] bytes = greeting.getBytes(transformOptions.get("sourceEncoding"));
outputStream.write(bytes, 0, bytes.length);
}
@Override
public ProbeTestTransform getProbeTestTransform()
private String getGreeting(String language)
{
// See the Javadoc on this method and Probes.md for the choice of these values.
return new ProbeTestTransform(this, "quick.pdf", "quick.png",
7455, 1024, 150, 10240, 60 * 20 + 1, 60 * 15 - 15)
{
@Override
protected void executeTransformCommand(File sourceFile, File targetFile)
{
transformImpl(null, null, null, Collections.emptyMap(), sourceFile, targetFile);
}
};
return "Hello %s";
}
@Override
public void transformImpl(String transformName, String sourceMimetype, String targetMimetype,
Map<String, String> transformOptions, File sourceFile, File targetFile)
{
executor.transform(sourceMimetype, targetMimetype, transformOptions, sourceFile, targetFile);
}
}
~~~
* *TransformerName*Executer.java - *JavaExecutor* and *CommandExecutor* sub classes need to extract values from
*transformOptions* and use them in a call to an external process or as parameters to a library call.
~~~
...
public class TransformerNameExecutor extends AbstractCommandExecutor
{
...
@Override
public void transform(String transformName, String sourceMimetype, String targetMimetype,
Map<String, String> transformOptions,
File sourceFile, File targetFile) throws TransformException
{
final String options = TransformerNameOptionsBuilder
.builder()
.withWidth(transformOptions.get(WIDTH_REQUEST_PARAM))
.withHeight(transformOptions.get(HEIGHT_REQUEST_PARAM))
.build();
Long timeout = stringToLong(transformOptions.get(TIMEOUT));
run(options, sourceFile, targetFile, timeout);
}
}
~~~
* Application.java - [Spring Boot](https://projects.spring.io/spring-boot/) expects to find an Application in
a project's source files. The following may be used:
~~~
package org.alfresco.transformer;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class Application
{
public static void main(String[] args)
{
SpringApplication.run(Application.class, args);
}
}
~~~
Transform requests are handled by the *TransformController*, but are either:
* POST requests (a direct http request from a client) where the transform options are passed as parameters, the source is supplied as a multipart file and
the response is a file download.
* POST request (a request via a message queue) where the transform options are supplied as JSON and the response is also JSON.
The source and target content is read from a location accessible to both the client and the transfomer.
**Example JSON request body**
```javascript
var transformRequest = {
"requestId": "1",
"sourceReference": "2f9ed237-c734-4366-8c8b-6001819169a4",
"sourceMediaType": "application/pdf",
"sourceSize": 123456,
"sourceExtension": "pdf",
"targetMediaType": "text/plain",
"targetExtension": "txt",
"clientType": "ACS",
"clientData": "Yo No Soy Marinero, Soy Capitan, Soy Capitan!",
"schema": 1,
"transformRequestOptions": {
"targetMimetype": "text/plain",
"targetEncoding": "UTF-8",
"abc:width": "120",
"abc:height": "200"
}
}
```
**Example JSON response body**
```javascript
var transformReply = {
"requestId": "1",
"status": 201,
"errorDetails": null,
"sourceReference": "2f9ed237-c734-4366-8c8b-6001819169a4",
"targetReference": "34d69ff0-7eaa-4741-8a9f-e1915e6995bf",
"clientType": "ACS",
"clientData": "Yo No Soy Marinero, Soy Capitan, Soy Capitan!",
"schema": 1
**Example T-Config** `resources/hello_engine_config.json`
```json
{
"transformOptions": {
"helloOptions": [
{"value": {"name": "language"}},
{"value": {"name": "sourceEncoding"}}
]
},
"transformers": [
{
"transformerName": "hello",
"supportedSourceAndTargetList": [
{"sourceMediaType": "text/plain", "targetMediaType": "text/plain" }
],
"transformOptions": [
"helloOptions"
]
}
]
}
```
## Building and testing
The project can be built by running the Maven command:
~~~
mvn clean install
~~~
## Artifacts
The artifacts can be obtained by:
* downloading from the [Alfresco repository](https://artifacts.alfresco.com/nexus/content/groups/public/)
* Adding a Maven dependency to your pom file.
~~~
<dependency>
<groupId>org.alfresco</groupId>
<artifactId>alfresco-base-t-engine</artifactId>
<version>1.0</version>
</dependency>
~~~
and the Alfresco Maven repository:
~~~
<repository>
<id>alfresco-maven-repo</id>
<url>https://artifacts.alfresco.com/nexus/content/groups/public</url>
</repository>
~~~
The build plan is available in [TravisCI](https://travis-ci.com/Alfresco/alfresco-transform-core).
## Contributing guide
Please use [this guide](https://github.com/Alfresco/alfresco-repository/blob/master/CONTRIBUTING.md)
to make a contribution to the project.
**Example `ProbeTransform` test file** `jane.txt`
```json
Jane
```

4
engines/base/src/main/java/org/alfresco/transform/base/TransformEngine.java
View File

@@ -50,7 +50,9 @@ public interface TransformEngine
/**
* @return a definition of what the t-engine supports. Normally read from a json Resource on the classpath using a
* {@link TransformConfigResourceReader}.
* {@link TransformConfigResourceReader}. To combine to code from multiple t-engine into a single t-engine
* include all the TransformEngines and CustomTransform implementations, plus a wrapper TransformEngine for the
* others. The wrapper should return {@code null} from this method.
*/
TransformConfig getTransformConfig();

6
engines/base/src/main/java/org/alfresco/transform/base/TransformHandler.java
View File

@@ -74,6 +74,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;
import static java.util.stream.Collectors.joining;
import static org.alfresco.transform.base.fs.FileManager.createTargetFile;
@@ -638,7 +639,10 @@ public class TransformHandler
sourceSizeInBytes, targetMimetype, transformOptions, null);
if (transformerName == null)
{
throw new TransformException(BAD_REQUEST, "No transforms were able to handle the request");
throw new TransformException(BAD_REQUEST, "No transforms were able to handle the request: "+
sourceMimetype+" -> "+targetMimetype+transformOptions.entrySet().stream()
.map(entry -> entry.getKey()+"="+entry.getValue())
.collect(Collectors.joining(", ", " with ", "")));
}
return transformerName;
}

17
engines/base/src/main/java/org/alfresco/transform/base/probes/ProbeTransform.java
View File

@@ -51,9 +51,9 @@ import static org.springframework.http.HttpStatus.TOO_MANY_REQUESTS;
/**
* Provides test transformations and the logic used by k8 liveness and readiness probes.
*
* <p><b>K8s probes</b>: A readiness probe indicates if the pod should accept request. <b>It does not indicate that a pod is
* ready after startup</b>. The liveness probe indicates when to kill the pod. <b>Both probes are called throughout the
* lifetime of the pod</b> and a <b>liveness probes can take place before a readiness probe.</b> The k8s
* <p><b>K8s probes</b>: A readiness probe indicates if the pod should accept request. <b>It does not indicate that a
* pod is ready after startup</b>. The liveness probe indicates when to kill the pod. <b>Both probes are called
* throughout the lifetime of the pod</b> and a <b>liveness probes can take place before a readiness probe.</b> The k8s
* <b>initialDelaySeconds field is not fully honoured</b> as it is multiplied by a random number, so is
* actually a maximum initial delay in seconds, but could be 0. </p>
*
@@ -81,7 +81,6 @@ public class ProbeTransform
private static final int AVERAGE_OVER_TRANSFORMS = 5;
private final String sourceFilename;
private final String targetFilename;
private final String sourceMimetype;
private final String targetMimetype;
private final Map<String, String> transformOptions;
@@ -115,13 +114,11 @@ public class ProbeTransform
return maxTime;
}
public ProbeTransform(String sourceFilename, String targetFilename,
String sourceMimetype, String targetMimetype, Map<String, String> transformOptions,
public ProbeTransform(String sourceFilename, String sourceMimetype, String targetMimetype, Map<String, String> transformOptions,
long expectedLength, long plusOrMinus, int livenessPercent, long maxTransforms, long maxTransformSeconds,
long livenessTransformPeriodSeconds)
{
this.sourceFilename = sourceFilename;
this.targetFilename = targetFilename;
this.sourceMimetype = sourceMimetype;
this.targetMimetype = targetMimetype;
this.transformOptions = new HashMap(transformOptions);
@@ -271,14 +268,14 @@ public class ProbeTransform
getMessagePrefix(isLiveProbe) + "Failed to store the source file", e);
}
long length = sourceFile.length();
LogEntry.setSource(sourceFilename, length);
LogEntry.setSource(sourceFile.getName(), length);
return sourceFile;
}
private File getTargetFile()
{
File targetFile = createTempFile("target_", "_" + targetFilename);
LogEntry.setTarget(targetFilename);
File targetFile = createTempFile("target_", "_" + sourceFilename);
LogEntry.setTarget(targetFile.getName());
return targetFile;
}

1
engines/base/src/main/resources/engine_config.json
View File

@@ -1 +0,0 @@
{}

15
engines/base/src/test/java/org/alfresco/transform/base/QueueTransformServiceTest.java
View File

@@ -46,11 +46,14 @@ import javax.jms.Destination;
import javax.jms.JMSException;
import javax.jms.Message;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doCallRealMethod;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoMoreInteractions;
import static org.mockito.Mockito.when;
import static org.springframework.http.HttpStatus.BAD_REQUEST;
import static org.springframework.http.HttpStatus.CREATED;
import static org.springframework.http.HttpStatus.INTERNAL_SERVER_ERROR;
@@ -187,13 +190,13 @@ public class QueueTransformServiceTest
.build();
doReturn(request).when(transformMessageConverter).fromMessage(msg);
doReturn(new ResponseEntity<>(reply, HttpStatus.valueOf(reply.getStatus())))
.when(transformHandler).handleMessageRequest(request, null, null);
doAnswer(invocation -> {transformReplySender.send(destination, reply); return null;})
.when(transformHandler).handleMessageRequest(request, null, destination);
queueTransformService.receive(msg);
verify(transformMessageConverter).fromMessage(msg);
verify(transformHandler).handleMessageRequest(request, null, null);
verify(transformHandler).handleMessageRequest(request, null, destination);
verify(transformReplySender).send(destination, reply);
}
@@ -228,13 +231,13 @@ public class QueueTransformServiceTest
.build();
doReturn(request).when(transformMessageConverter).fromMessage(msg);
doReturn(new ResponseEntity<>(reply, HttpStatus.valueOf(reply.getStatus())))
.when(transformHandler).handleMessageRequest(request, null, null);
doAnswer(invocation -> {transformReplySender.send(destination, reply); return null;})
.when(transformHandler).handleMessageRequest(request, null, destination);
queueTransformService.receive(msg);
verify(transformMessageConverter).fromMessage(msg);
verify(transformHandler).handleMessageRequest(request, null, null);
verify(transformHandler).handleMessageRequest(request, null, destination);
verify(transformReplySender).send(destination, reply);
}
}

13
engines/base/src/test/java/org/alfresco/transform/base/fakes/FakeTransformEngineWithTwoCustomTransformers.java
View File

@@ -46,7 +46,8 @@ import static org.alfresco.transform.common.RequestParamMap.SOURCE_ENCODING;
public class FakeTransformEngineWithTwoCustomTransformers extends AbstractFakeTransformEngine
{
@Override public TransformConfig getTransformConfig()
@Override
public TransformConfig getTransformConfig()
{
String docOptions = "docOptions";
String imageOptions = "imageOptions";
@@ -103,11 +104,11 @@ public class FakeTransformEngineWithTwoCustomTransformers extends AbstractFakeTr
.build();
}
@Override public ProbeTransform getProbeTransform()
@Override
public ProbeTransform getProbeTransform()
{
return new ProbeTransform("quick.txt", "quick.pdf",
MIMETYPE_TEXT_PLAIN, MIMETYPE_PDF, ImmutableMap.of(SOURCE_ENCODING, "UTF-8"),
46, 0, 150, 1024, 1,
60 * 2);
return new ProbeTransform("quick.txt", MIMETYPE_TEXT_PLAIN, MIMETYPE_PDF,
ImmutableMap.of(SOURCE_ENCODING, "UTF-8"), 46, 0,
150, 1024, 1, 60 * 2);
}
}