mirror of
https://github.com/Alfresco/SearchServices.git
synced 2026-09-16 18:12:56 +00:00
Merge master into org.alfresco-alfresco-data-model-8.32.
This commit is contained in:
Tom Page
@@ -0,0 +1,46 @@
|
||||
# 3. Combined Codebase
|
||||
|
||||
Date: 09/04/2019
|
||||
|
||||
## Status
|
||||
|
||||
Approved
|
||||
|
||||
## Context
|
||||
|
||||
Historically Alfresco has had a bad experience of having a monolithic codebase in SVN. The main issue with this was the
|
||||
inability to easily work on feature branches, resulting in frequent conflicting changes. To make matters worse this was
|
||||
seen as hard to move to git, since GitHub has/had a maximum limit on repository size and there were several large
|
||||
binary artifacts stored in the Alfresco history [[1]].
|
||||
|
||||
More recently the Alfresco codebase has been split into a large number of small git repositories [[2]].
|
||||
|
||||
This proposal is to merge the various git repositories together. By doing this we aim to:
|
||||
|
||||
1. Remove the effort of creating releases of library projects.
|
||||
1. Allow tests to be written on the same branch (in the same repository) as the production code (i.e. facilitate TDD).
|
||||
1. Make it easier for everyone (especially people outside the team and new starters) to find the 'right bit' of code.
|
||||
|
||||
[1]: https://community.alfresco.com/community/ecm/blog/2015/04/01/so-when-is-alfresco-moving-to-github
|
||||
|
||||
[2]: https://ts.alfresco.com/share/proxy/alfresco/api/node/content/versionStore/version2Store/a0c2492f-6354-4b98-adfc-e63d5c2209f5/SearchCodeBase.png
|
||||
|
||||
## Decision
|
||||
|
||||
We will merge the search-related repositories together and preserve their history. We will not attempt to merge code
|
||||
that other teams also need (for example the TAS test utilities or the alfresco-data-model projects).
|
||||
|
||||
## Consequences
|
||||
|
||||
An epic has been raised containing the next steps [[3]]. Broadly speaking these are:
|
||||
|
||||
1. Merge the alfresco-solr-client library with the SearchServices repository.
|
||||
1. Merge the insight-engine, insight-jdbc and insight-zeppelin repositories.
|
||||
1. A consequence of this is that future releases of these artifacts will share version numbers.
|
||||
1. Put all the end-to-end tests in a single testing project.
|
||||
1. Merge the (public) SearchServices code into the (private) InsightEngine repository and set up a mirror so the
|
||||
community code is still publicly editable.
|
||||
1. Merge the tests with the production code.
|
||||
1. Merge the build scripts with the production code.
|
||||
|
||||
[3]: https://issues.alfresco.com/jira/browse/SEARCH-1393
|
||||
@@ -0,0 +1,44 @@
|
||||
# 4. Community Mirror
|
||||
|
||||
Date: 09/04/2019
|
||||
|
||||
## Status
|
||||
|
||||
Approved
|
||||
|
||||
## Context
|
||||
|
||||
In [ADR 3: "Combined Codebase"](0003-combined-codebase.md) we decided to merge the Search Services and Insight Engine
|
||||
repositories. Since we want to enable the community to submit pull requests to the Search Services project we need a
|
||||
way to keep this code up to date on GitHub.
|
||||
|
||||
## Decision
|
||||
|
||||
We will mirror `master` and all branches starting with `release/` to a branch with the same name on GitHub. We will
|
||||
exclude the alfresco-insight-engine-parent directory. We will include these commands as part of our build to do this:
|
||||
|
||||
```
|
||||
# This avoids making changes to the original branch.
|
||||
get checkout -b tempBranch
|
||||
# This strips all enterprise changes (in a reproducible way) and pushes any updates to the mirror.
|
||||
git filter-branch -f --prune-empty --index-filter 'git rm -r --cached --ignore-unmatch alfresco-insight-engine-parent'
|
||||
git push out HEAD:$branch
|
||||
# This resets us back to where we were before the filtering.
|
||||
git checkout $branch
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
A ticket has been raised to track this [[1]]. A script has been written to do the initial codebase merge and
|
||||
mirroring [[2]].
|
||||
|
||||
We will rewrite the whole history of the SearchServices repository using the mirroring command. This will result in the
|
||||
new repository containing a new root pom file and the search services code beneath that.
|
||||
|
||||
All changes except those within the module called insight-engine will be mirrored.
|
||||
|
||||
The root pom file in the community project will include a reference to the insight-engine pom file, so community users
|
||||
will have to build from within the alfresco-search-parent directory.
|
||||
|
||||
[1]: https://issues.alfresco.com/jira/browse/SEARCH-1397
|
||||
[2]: https://git.alfresco.com/search_discovery/combinerScript/blob/master/combineSearch.sh
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
# 5. Merge Tests And Production Code
|
||||
|
||||
Date: 09/04/2019
|
||||
|
||||
## Status
|
||||
|
||||
Approved
|
||||
|
||||
## Context
|
||||
|
||||
In [ADR 3: "Combined Codebase"](0003-combined-codebase.md) we decided to merge the production and end-to-end test
|
||||
repositories. In [ADR 4: "Community Mirror"](0004-community-mirror.md) we discuss setting up a mirror for the community
|
||||
code.
|
||||
|
||||
## Decision
|
||||
|
||||
We will separate the end-to-end test code in half so that any code solely related to Insight Engine won't be mirrored.
|
||||
|
||||
We will remove the existing test groups for the different versions of Search Services and Insight Engine, and instead
|
||||
delete any tests from branches where they should not be run.
|
||||
|
||||
## Consequences
|
||||
|
||||
It will be possible to include production code changes along with all required test changes in the same merge request.
|
||||
It will be easy to get new tests running before production code is written without causing other branches to fail.
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
# 6. Release Synchronisation
|
||||
|
||||
Date: 24/04/2019
|
||||
|
||||
## Status
|
||||
|
||||
Approved
|
||||
|
||||
## Context
|
||||
|
||||
In [ADR 3: "Combined Codebase"](0003-combined-codebase.md) we decided to merge the Search Services and Insight Engine
|
||||
repositories. In [ADR 4: "Community Mirror"](0004-community-mirror.md) we discussed how we would set up a build job to
|
||||
ensure community code is available to the community. In particular we decided:
|
||||
|
||||
> We will mirror `master` and all branches starting with `release/` to a branch with the same name on GitHub.
|
||||
|
||||
## Decision
|
||||
|
||||
During the combining of the Search Services and Insight Engine codebases we will create a branch `master` by merging
|
||||
the existing `master` branches of those two projects. We will update the version of Insight Engine to match Search Services - that
|
||||
is 1.4.0-SNAPSHOT.
|
||||
|
||||
## Consequences
|
||||
|
||||
The next minor release of Insight Engine will be 1.4.0 which will happen at the same time as the release of Search
|
||||
Services 1.4.0. There will be no Insight Engine releases with the versions 1.2.x or 1.3.x.
|
||||
|
||||
Here are a few examples to illustrate the process for future releases. Note that other releases (including service
|
||||
packs, etc.) will follow a similar logic based on whether they are pre- or post- 1.4.0.
|
||||
|
||||
* SS 1.3.0 needs a hotfix
|
||||
|
||||
We release from the `release/alfresco-search-services/V1.3.0.x` branch.
|
||||
|
||||
* IE 1.1.0 needs a hotfix
|
||||
|
||||
We release from the `release/alfresco-insight-engine/V1.1.0.x` branch. We need to use the maven version 1.1.0.2 for the
|
||||
IE modules and 1.4.0-IE1.1.0.2 for the SS modules.
|
||||
|
||||
* SS 1.4.0 needs a hotfix
|
||||
|
||||
We'll create a hotfix release for both SS and IE (since they will both have the same issue). This will be from the
|
||||
`release/V1.4.0.x` branch (which would have already been created if SS/IE 1.4.0 had been released).
|
||||
|
||||
* IE 1.4.0 needs a hotfix
|
||||
|
||||
We'll create a hotfix release for both SS and IE from `release/V1.4.0.x` (and not publicise SS 1.4.0.1).
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 114 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 100 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 141 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 70 KiB |
+314
@@ -0,0 +1,314 @@
|
||||
# Event Based Content Tracker for Alfresco
|
||||
|
||||
|
||||
|
||||
**Contents**
|
||||
|
||||
- [Context](#context)
|
||||
- [How Text Extraction Works](#how-text-extraction-works)
|
||||
- [How the New Transform Service Works](#how-the-new-transform-service-works)
|
||||
- [Alternatives evaluation](#alternatives-evaluation)
|
||||
- [1 - No Change to Search Service](#1---no-change-to-search-service)
|
||||
- [2 - New Microservice in the Middle](#2---new-microservice-in-the-middle)
|
||||
- [3 - Event Oriented Content Tracking](#3---event-oriented-content-tracking)
|
||||
- [Findings](#findings)
|
||||
- [References](#references)
|
||||
|
||||
## Context
|
||||
|
||||
The current approach of the Content Tracker in Search Service is to query Solr for any *dirty* documents which it then fetches from Alfresco. Once the content is successfully obtained from Alfresco, it marks it clean which eventually get committed to the index. This approach will need to be modified as it applies pressure on Alfresco on every call to get the content. Taking an event based approach where the Content Tracker subscribes to a *topic* with policy the specific behaviour will allow to get the extracted content when ready.
|
||||
|
||||
There are currently different alternatives to be considered:
|
||||
|
||||
1. No change to Search Service. The repository will off load the transformation to Transform Service and pause the thread requesting the transformation until the content is transformed.
|
||||
2. Introduce a new microservice that sits between Search Service and the Transform Service. The content is off loaded to Transform Service and the microservice stores the content for Search Service to ingest.
|
||||
3. The Content Tracker that consumes content based on events
|
||||
|
||||
This document describes and explores these alternatives to provide the background required to produce an **ADR** (Architectural Decision Record) related to this topic.
|
||||
|
||||
|
||||
## How Text Extraction Works
|
||||
|
||||
Following information has been extracted from Alfresco Content Services 6.1 and Alfresco Search Services 1.3.
|
||||
|
||||
|
||||
|
||||
**Components referenced in the figure**
|
||||
|
||||
* [alfresco-search](https://github.com/Alfresco/SearchServices/tree/master/alfresco-search): Alfresco SOLR Application, deployed in SOLR Jetty web application `solr.war` as library
|
||||
* [alfresco-remote-api](https://github.com/Alfresco/alfresco-remote-api): REST API Layer for Alfresco Content Services, deployed in Alfresco Tomcat web application `alfresco.war` as library
|
||||
* [alfresco-repository](https://github.com/Alfresco/alfresco-repository): Metadata and content storage for Alfresco Content Services, deployed in Alfresco Tomcat web application `alfresco.war` as library
|
||||
* [Apache Tika](https://github.com/Alfresco/alfresco-tika): Detection and extraction of metadata and text from content, deployed in Alfresco Tomcat web application `alfresco.war` as library
|
||||
* [LibreOffice](https://www.libreoffice.org): This program is used in *serverless* mode to provide content transformation, installed locally on a server. It can be used *DIRECT* as TCP service (usually listening in port 8100) or via [JodConverter](http://docs.alfresco.com/community/concepts/OOo-subsystems-intro.html).
|
||||
|
||||
**Text Extraction Process**
|
||||
|
||||
Since more detailed information on the Content Tracker [1] is available, the text extraction process happens inside `SolrInformationServer` class. When *dirty* documents are being indexed, `SOLRAPIClient` is invoked (1) to retrieve Text Content from Alfresco Repository using the REST API (2)
|
||||
|
||||
```
|
||||
/api/solr/textContent?nodeId=<id>&propertyQName=<name>
|
||||
```
|
||||
|
||||
* `id` is the identifier of the document in database
|
||||
* `propertyQName` is the identifier of the property representing the content of the document, `{http://www.alfresco.org/model/content/1.0}content` by default
|
||||
|
||||
Both classes are part of [Alfresco Solr Application](https://github.com/Alfresco/SearchServices/tree/master/alfresco-search).
|
||||
|
||||
The request to `textContent` is processed by `NodeContentGet` Web Script (3) at [Alfresco Remote API](https://github.com/Alfresco/alfresco-remote-api). This component finds the right Content Transformer to produce the required Text Content from the original mime type. *Apache Tika*, as local class invocation, and *LibreOffice*, as TCP service or using JodConverter [2], are the Content Transformers available.
|
||||
|
||||
Once the transformation has been performed (4) by the Content Transformer, the Stream is returned back to `SolrInformationServer` (5) and the indexing process continues.
|
||||
|
||||
*Note*: `NodeContentGet` web script is using a deprecated API to find the right transformer: [getActiveTransformers()](https://github.com/Alfresco/alfresco-repository/blob/alfresco-repository-7.60/src/main/java/org/alfresco/repo/content/ContentServiceImpl.java#L897).
|
||||
|
||||
|
||||
## How the New Transform Service Works
|
||||
|
||||
Before going on with the alternatives evaluation, a quick overview of the new **Alfresco Transform Service** is provided.
|
||||
|
||||
Since ACS 6.1, transformations are performed by different *microservices* named as *Transform Engines* (T-Engine):
|
||||
|
||||
* [alfresco-docker-alfresco-pdf-renderer](https://git.alfresco.com/Repository/alfresco-docker-transformers/tree/master/alfresco-docker-alfresco-pdf-renderer) includes a PDFium [3] *wrapper* named Alfresco PDF Renderer [4] to provide operations on PDF files, as extracting the first page
|
||||
* [alfresco-docker-imagemagick](https://git.alfresco.com/Repository/alfresco-docker-transformers/tree/master/alfresco-docker-imagemagick) includes an ImageMagick [5] *wrapper* to provide operations on images, as generating a thumbnail
|
||||
* [alfresco-docker-libreoffice](https://git.alfresco.com/Repository/alfresco-docker-transformers/tree/master/alfresco-docker-libreoffice) includes a LibreOffice *wrapper* to provide transformation operations from a source mime type to a target mime type
|
||||
* [alfresco-docker-tika](https://git.alfresco.com/Repository/alfresco-docker-transformers/tree/master/alfresco-docker-tika) exposes Apache Tika [6] features for mime type transformation
|
||||
|
||||
>> These components are only available for Enterprise releases of Alfresco, so links in the text may be unavailable to Community users.
|
||||
|
||||
A *Transform Engine* is a dockerized REST API providing Transform Services and *renderization* operations. The [Transform Service](https://github.com/Alfresco/alfresco-transform-service) is a message router, currently working with ActiveMQ, to provide Transformation Requests to Transform Engines and to manage Transformation Replies from Transform Engines. This *router* finds the right Transform Engine for a Transformation Request depending on the source and target mime types.
|
||||
|
||||
To reduce the transit of content between the services, a dockerized [Shared File Store](https://github.com/Alfresco/alfresco-shared-file-store) REST API is used to have a single point of storage. When a transformation operation is requested, the source file is uploaded to the *Shared File Store* and when the transformation operation has been performed, the target file is uploaded also to that *Shared File Store*. Once the operation has been consumed, both source and target files are removed from *Shared File Store*.
|
||||
|
||||
|
||||
|
||||
**Components referenced in the figure**
|
||||
|
||||
* [T-Client](https://github.com/Alfresco/alfresco-repository/blob/alfresco-repository-7.60/src/main/java/org/alfresco/messaging/camel/routes/OnContentUpdateRenditionRoute.java#L88): Currently ACS is a transformer client, requesting transformation operations to *Transform Service* via JMS events
|
||||
* [Shared File Store](https://github.com/Alfresco/alfresco-shared-file-store): REST API Layer to share contents between a *T-Client* and the *T-Engine*
|
||||
* [T-Service](https://github.com/Alfresco/alfresco-transform-service): Alfresco Transform Service to route JMS Events to T-Engines
|
||||
* [T-Engine](https://git.alfresco.com/Repository/alfresco-docker-transformers): Transformation Engines provides transformation operations on content, consuming the source from `Shared File Store` and producing the target also in `Shared File Store`. Currently Apache Tika, LibreOffice, ImageMagick and PDF Renderer are available
|
||||
|
||||
>> T-Engine is currently only available for Enterprise releases of Alfresco, so links in the text may be unavailable to Community users.
|
||||
|
||||
**Transformation Process**
|
||||
|
||||
*T-Client* uploads the source content to `Shared File Store` using a REST API invocation (1) to retrieve a *srcContentRef* (2).
|
||||
|
||||
After that, *T-Client* creates a new `TransformRequest` event with the *srcContentRef* and the requested operation (3).
|
||||
|
||||
*T-Service* consumes the event and invokes to the right *T-Engine* (LibreOffice, ImageMagick, PdfRenderer, Apache Tika) with the *srcContentRef* and the operation requested (4).
|
||||
|
||||
The *T-Engine* performs the transformation (5) and stores the result in the `Shared File Store` to retrieve a *targetContentRef* (6).
|
||||
|
||||
*T-Service* creates a new `TransformReply` event with the *targetContentRef* that is consumed by *T-Client* (7), where the transformed content is downloaded from `Shared File Store` using a REST API invocation (8). After that, for instance, this downloaded content is stored in the repository as a new *rendition*, so the temporary copy at `Shared File Store` is not required anymore.
|
||||
|
||||
Both source and target files can be deleted from `Shared File Store` after the operation ends successfully (9).
|
||||
|
||||
*Note*. Detailed JSON syntax for `Transform Request` and `Transform Reply` are available at [Spec for Transform Request & Reply](https://github.com/Alfresco/alfresco-transform-service/blob/master/docs/transformation-request-reply.md)
|
||||
|
||||
|
||||
## Alternatives evaluation
|
||||
|
||||
Below different alternatives for integrating Content Tracking with the New Transform Service are described.
|
||||
|
||||
|
||||
### 1 - No Change to Search Service
|
||||
|
||||
The repository will off load the transformation to Transform Service and pause the thread requesting the transformation until the content is transformed.
|
||||
|
||||
|
||||
|
||||
**Components referenced in the figure**
|
||||
|
||||
* [alfresco-search](https://github.com/Alfresco/SearchServices/tree/master/alfresco-search): Alfresco SOLR Application, deployed in SOLR Jetty web application `solr.war` as library
|
||||
* [alfresco-remote-api](https://github.com/Alfresco/alfresco-remote-api): REST API Layer for Alfresco Content Services, deployed in Alfresco Tomcat web application `alfresco.war` as library
|
||||
* [alfresco-repository](https://github.com/Alfresco/alfresco-repository): Metadata and content storage for Alfresco Content Services, deployed in Alfresco Tomcat web application `alfresco.war` as library
|
||||
* [Shared File Store](https://github.com/Alfresco/alfresco-shared-file-store): REST API Layer to share contents between a *T-Client* and the *T-Engine*
|
||||
* [T-Service](https://github.com/Alfresco/alfresco-transform-service): Alfresco Transform Service to route JMS Events to T-Engines
|
||||
* [DockerTika](https://git.alfresco.com/Repository/alfresco-docker-transformers/tree/master/alfresco-docker-tika) REST API Layer for *T-Engine* based in Tika
|
||||
* [DockerLibreoffice](https://git.alfresco.com/Repository/alfresco-docker-transformers/tree/master/alfresco-docker-libreoffice) REST API Layer for *T-Engine* based in LibreOffice
|
||||
|
||||
>> T-Engine components, like Tika and LibreOffice, are only available for Enterprise releases of Alfresco, so links in the text may be unavailable to Community users.
|
||||
|
||||
**Transformation Process**
|
||||
|
||||
From SOLR side, the code remains unchanged, so the transformation to text is requested to `NodeContentGet` web script (2) as before.
|
||||
|
||||
From this point, *Content Transformer* stores the source content in the `Shared File Store` to retrieve the `sourceRef` (4) and creates a new *Transform Request* event to be consumed by *Transform Service* (5).
|
||||
|
||||
*Transform Service* uses routing component to find the right *Transform Engine* (Tika, LibreOffice...) and perform the invocation with the `sourceRef` and the target media type (TXT) to the *Transform Engine* (6).
|
||||
|
||||
Source Content is retrieved from `Shared File Store` using `sourceRef` (7) and the transformation is performed (8) and the result is stored in the `Shared File Store` to retrieve the `targetRef` (9).
|
||||
|
||||
After that, *Transform Engine* creates a new *Transform Reply* event (10) to be consumed by *Content Transformer*, where the content is retrieved from `Shared File Store` using `targetRef` (11).
|
||||
|
||||
As the original request to `NodeContentGet` has been paused while performing the transformation, *Content Transformer* returns the TextContent Stream back and the response is delivered to SOLR (12).
|
||||
|
||||
Both source and target files can be deleted from `Shared File Store` after the operation ends successfully (13,14).
|
||||
|
||||
**Consequences**
|
||||
|
||||
No consequences for *Search Services* are expected from applying this alternative.
|
||||
|
||||
|
||||
### 2 - New Microservice in the Middle
|
||||
|
||||
Introduce a new microservice that sits between Search Service and the Transform Service. The content is off loaded to Transform Service and the microservice stores the content for Search Service to ingest.
|
||||
|
||||
|
||||
|
||||
**Components referenced in the figure**
|
||||
|
||||
* [alfresco-search](https://github.com/Alfresco/SearchServices/tree/master/alfresco-search): Alfresco SOLR Application, deployed in SOLR Jetty web application `solr.war` as library
|
||||
* *ContentTracker*: New REST API Layer based in Spring Boot App (to be dockerizable) to be designed. This component is also publishing `TransformRequest` events to be consumed by *Transform Service*
|
||||
* [alfresco-remote-api](https://github.com/Alfresco/alfresco-remote-api): REST API Layer for Alfresco Content Services, deployed in Alfresco Tomcat web application `alfresco.war` as library
|
||||
* [Shared File Store](https://github.com/Alfresco/alfresco-shared-file-store): REST API Layer to share contents between a *T-Client* and the *T-Engine*
|
||||
* [T-Service](https://github.com/Alfresco/alfresco-transform-service): Alfresco Transform Service to route JMS Events to T-Engines
|
||||
* [DockerTika](https://git.alfresco.com/Repository/alfresco-docker-transformers/tree/master/alfresco-docker-tika) REST API Layer for *T-Engine* based in Tika
|
||||
* [DockerLibreoffice](https://git.alfresco.com/Repository/alfresco-docker-transformers/tree/master/alfresco-docker-libreoffice) REST API Layer for *T-Engine* based in LibreOffice
|
||||
|
||||
>> T-Engines, like Tika and LibreOffice, are only available for Enterprise releases of Alfresco, so links in the text may be unavailable to Community users.
|
||||
|
||||
**Transformation Process**
|
||||
|
||||
>> A new microservice, named *ContentTracker* in the diagram, will handle transformations to text requests from Content Tracker. This service replaces previous `NodeContentGet` web script.
|
||||
|
||||
SolrInformationServer invokes to *ContentTracker* using HTTP REST API to start the transformation (1).
|
||||
|
||||
*ContentTracker* invokes the repository to retrieve the source content for the node using the Remote API (2) and stores the result in the `Shared File Store` to retrieve the `sourceRef` (3).
|
||||
|
||||
After that, *ContentTracker* creates a new *Transform Request* event to be consumed by *Transform Service* (4).
|
||||
|
||||
*Transform Service* uses routing component to find the right *Transform Engine* (Tika, LibreOffice...) and perform the invocation with the `sourceRef` and the target media type (TXT) to the *Transform Engine* (5).
|
||||
|
||||
Source Content is retrieved from `Shared File Store` using `sourceRef` (6) and the transformation is performed (7) and the result is stored in the `Shared File Store` to retrieve the `targetRef` (8).
|
||||
|
||||
After that, *Transform Engine* creates a new *Transform Reply* event (9) to be consumed by *ContentTracker*, where the content is retrieved from `Shared File Store` using `targetRef` (10).
|
||||
|
||||
As the original request (1) has been paused while performing the transformation, *ContentTracker* returns the TextContent Stream back and the response is delivered to Content Tracker (11).
|
||||
|
||||
Both source and target files can be deleted from `Shared File Store` after the operation ends successfully (12,13).
|
||||
|
||||
**Alternative implementation**
|
||||
|
||||
A variation of this scenario is to adapt the indexing flow for the transform of content "to text" to be async by using `Transform Request` / `Transform Reply`.
|
||||
|
||||
*Search Service* requests Repo to transfer content into *Shared File Store* and then *Search Services* sends the async transform request to the new Transform Service. The *Search Service* then consumes the async Transform Reply (unique for each shard) in order to update the index shard.
|
||||
|
||||
The main difference with original option is that *ContentTracker* does not get the original content and stores it in *Shared File Store* but sends a request to *Repository* to do that, what is better in terms of performance.
|
||||
|
||||
**Consequences**
|
||||
|
||||
* A new dependency from Transform Service version must be included in Search Services compatibility matrix.
|
||||
* It's required to create a switch in Search Services to use current Content Tracker (based in REST API invocation) or the new Content Tracker (including JMS events), mainly to preserve 5.2 compatibility.
|
||||
* A new component *ContentTraker* must be developed. This component is a new *T-Client*, dockerizable and JMS ready. Incoming threads must be paused till the JMS *TransformReply* is consumed, so some logic and control need to be added to current code [7]
|
||||
* **RISK** Currently Transform Service is not supporting several *T-Clients* for the same *Transform Service*, despite the story is being developed [ATS-208](https://issues.alfresco.com/jira/browse/ATS-208). This will exclude this alternative, as *ACS* is the only *T-Client* supported by now. Also, for a Sharding environment, more than one indexing *T-Client* can be required.
|
||||
|
||||
|
||||
### 3 - Event Oriented Content Tracking
|
||||
|
||||
The Content Tracker that consumes content based on events.
|
||||
|
||||
**Why Apache Kafka**
|
||||
|
||||
SOLR indexes require a full history of events (transactions) to be rebuilt from scratch, so message brokers with no storage (like *ActiveMQ*, currently used by ACS and Transform Service) are not recommended for this scenario.
|
||||
|
||||
Since Amazon is providing an [MSK](https://aws.amazon.com/msk/) (Amazon Managed Streaming for Kafka) and Apache Kafka is a streaming data store, this product seems to fit better the requirements for Search Services.
|
||||
|
||||
Apache Kafka *PoC* has been developed in order to test different requirements:
|
||||
|
||||
* Apache Kafka and Zookeeper compatibility with current [ACS Stack](https://github.com/Alfresco/acs-deployment)
|
||||
|
||||
* Producing events to a topic using messages in JSON format
|
||||
|
||||
* Consuming events from a topic: live, from an offset and from the beginning
|
||||
|
||||
* Consuming events *live* from a topic is required for a regular running of the platform, where index requests are consumed to perform indexing operations
|
||||
|
||||
* Consuming events *from an offset* is required to provide a safe Search Service re-starting, as Indexing System needs then to catch up with events production that may happen during Search Service stopping
|
||||
|
||||
* Consuming events *from beginning* is required when rebuilding SOLR indexes from scratch
|
||||
|
||||
The *PoC* is available at https://github.com/aborroy/event-content-tracker-kafka
|
||||
|
||||
|
||||
**The I-x design**
|
||||
|
||||
The new paradigm introduced by *Kafka*, requires a set of components to be provided to the platform in order to provide the indexation process.
|
||||
|
||||
|
||||
|
||||
* *I-Client*: Spring Boot App receiving indexing requests from an ACS behaviour, a REST API invocation or an ActiveMQ message.
|
||||
* [Shared File Store](https://github.com/Alfresco/alfresco-shared-file-store): REST API Layer to store text files to be indexed.
|
||||
* *I-Service*: Spring Boot App listening to a *topic* in Apache Kafka to process `IndexRequest` messages.
|
||||
* *I-Engine*: Spring Boot App receiving REST API invocations to perform indexing operations.
|
||||
|
||||
The *I-Client* receives a new request for a node `Id` with `txtContentRef` to be indexed, and it produces a new `IndexRequest` message for Apache Kafka indexing `topic` (1)
|
||||
|
||||
The *I-Service* consumes the message from Apache Kafka indexing `topic` and invokes the *I-Engine* with the `Id` and the `txtContentRef` (2)
|
||||
|
||||
The *I-Engine* retrieves the text content file from *Shared File Store* (3) and performs the indexation in the indexer (SOLR) updating the current `offset` global value (4)
|
||||
|
||||
The result is propagated till the *I-Client* to provide feedback (5, 6)
|
||||
|
||||
|
||||
**Sample content tracking process**
|
||||
|
||||
|
||||
|
||||
**Components referenced in the figure**
|
||||
|
||||
* *OnUpdateBehaviour* [I-Client, T-Client]: This component can be developed as part of Alfresco Repository or as a new module for Alfresco Repository. This component is a *T-Client* (transformation client) and also a *I-Client* (indexation client). When the content of a node is updated, a new `TransformRequest` event to transform to text is published to ActiveMQ to be consumed by *Transform Service* and a new `IndexRequest` event to perform the indexation is published to Kafka Topic `Indexation`
|
||||
* [Shared File Store](https://github.com/Alfresco/alfresco-shared-file-store): REST API Layer to share contents between a *T-Client* and the *T-Engine*
|
||||
* [TransformService](https://github.com/Alfresco/alfresco-transform-service): Alfresco Transform Service to route JMS Events to *T-Engines*
|
||||
* *EventSearchService* [I-Service]: This component is a *I-Service*, receiving `IndexRequest` to be delivered to a *I-Engine*
|
||||
* *ContentTrackerService* [I-Engine]: This component is a *I-Engine*, providing the indexing logic for an `IndexRequest`
|
||||
* [alfresco-search](https://github.com/Alfresco/SearchServices/tree/master/alfresco-search): Custom SOLR Code including indexing logic components: `ContentTracker`, `SolrInformationServer` and `SolrContentStore`
|
||||
|
||||
**Transformation Process**
|
||||
|
||||
A new content updated event in the repository triggers a `TransformRequest` event to text to be consumed by *T-Service* (2) saving first the original file in *Shared File Store* (1)
|
||||
|
||||
When *T-Service* has performed the transformation, a `TransformReply` event is produced, that is consumed by *EventSearchService* to store the reference `targetRef` to the transformed content (3).
|
||||
|
||||
Also, after issuing the transformation event, a new `IndexRequest` event is triggered to Kafka Topic `Indexation` to be consumed by *EventSearchService* (4)
|
||||
|
||||
Once *EventSearchService* has a `TransformReply` and a matching `IndexRequest`, an indexation process is started by invoking *ContentTrackerService* REST API with the `targetRef` and the `txId` (5)
|
||||
|
||||
*ContentTrackerService* starts the indexation process for a `txId` by using current logic (6)
|
||||
|
||||
When reply is sent back to `I-Client`, current Kafka `offset` must be updated in `I-Engine` in order to be prepared for catching up the repository or re-building purposes (14)
|
||||
|
||||
**Consequences**
|
||||
|
||||
* A new dependency from Transform Service version must be included in Search Services compatibility matrix.
|
||||
* A new dependency from Apache Kafka version must be included in Search Services compatibility matrix.
|
||||
* It's required to create a switch in Search Services to use current Content Tracker (based in REST API invocation) or the new Content Tracker (including JMS events), mainly to preserve 5.2 compatibility.
|
||||
* A new component *EventSearchService* must be developed. This component is a new *I-Service*, dockerizable and JMS / Kafka ready.
|
||||
* A new component *ContentTrackerService* must be developed. This component is a new *I-Engine*, dockerizable and providing a REST API.
|
||||
* Since the new indexation repository is Apache Kafka, `Transaction Id` concept must be translated to Kafka transaction system, to provide the right *offset* to Consumers when re-indexing or catching up when indexation is behind Content Services updates.
|
||||
* **RISK** Currently Transform Service is not supporting several *T-Clients* for the same *Transform Service*, despite the story is being developed [ATS-208](https://issues.alfresco.com/jira/browse/ATS-208). This will exclude this alternative, as *ACS* is the only *T-Client* supported by now. Also, for a Sharding environment, more than one indexing *T-Client* can be required.
|
||||
|
||||
|
||||
## Findings
|
||||
|
||||
* Apache Kafka works inside ACS Deployment strategy, so it can be used alone or together with the one used by Transform Service (ApacheMQ).
|
||||
* An Apache Kafka topic is a feed name to which events are published. For each topic, the Kafka cluster maintains a partitioned log. Each partition is an ordered, immutable sequence of events that is continually appended to—a structured commit log. The records in the partitions are each assigned a sequential id number called the **offset** that uniquely identifies each record within the partition. So, this `offset` can be used by I-Engines to identify the order when recovering from a previous indexation status.
|
||||
* The Kafka cluster durably persists all published records whether or not they have been consumed. Kafka's performance is effectively constant with respect to data size so storing data for a long time (or even forever) is not a problem.
|
||||
* Apache Kafka is supported by Amazon, so it can be the right product to store indexing events also in Amazon deployments.
|
||||
* According to Apache Kafka, for better throughput, the max message size should be 10KB. So text content should not be stored within the event.
|
||||
|
||||
## References
|
||||
|
||||
[1] [Content Tracker](https://github.com/Alfresco/SearchServices/blob/master/alfresco-search/doc/architecture/trackers/00001-content-tracker.md)
|
||||
|
||||
[2] [JodConverter](https://github.com/Alfresco/alfresco-jodconverter)
|
||||
|
||||
[3] [PDFium](https://pdfium.googlesource.com/pdfium)
|
||||
|
||||
[4] [Alfresco PDF Renderer](https://git.alfresco.com/Repository/alfresco-pdf-renderer) - This source code is currently only available for Enteprise users.
|
||||
|
||||
[5] [ImageMagick](http://www.imagemagick.org)
|
||||
|
||||
[6] [Apache Tika](https://tika.apache.org)
|
||||
|
||||
[7] [PausableThreadPoolExecutor](https://gist.github.com/warmwaffles/8534618)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
+206
@@ -0,0 +1,206 @@
|
||||
## Tracker State
|
||||
|
||||
|
||||
|
||||
### Purpose and current implementation
|
||||
The _org.alfresco.solr.TrackerState_ class represents the state of a given tracker, with several information including the state of the tracker itself and some other data the system needs to remember about the ongoing tracking process.
|
||||
The state definition within the _TrackerState_ is actually the union of all attributes needed by the all trackers during their lifecycle.
|
||||
|
||||
The following table illustrates the member instances currently defined in the _TrackerState_, together with the indication of the tracker which makes use of that specific information.
|
||||
|
||||
| Property | READ | WRITE | Notes|
|
||||
| --------------------------|:----------:|------------------------------------|:------------|
|
||||
|lastChangeSetIdOnServer|ACLTracker|ACLTracker | |
|
||||
|lastChangeSetCommitTimeOnServer|ACLTracker|ACLTracker| |
|
||||
|lastIndexedChangeSetId*|ACLTracker, MetadataTracker::getShardState|ACLTracker| |
|
||||
|lastIndexedTxCommitTime|MetadataTracker|MetadataTracker| |
|
||||
|lastIndexedTxId|MetadataTracker|MetadataTracker| |
|
||||
|lastIndexedChangeSetCommitTime*|AclTracker, MetadataTracker, AbstractTracker::continueState|AclTracker| |
|
||||
|lastTxCommitTimeOnServer|MetadataTracker|N.A.| |
|
||||
|lastTxIdOnServer|MetadataTracker|MetadataTracker| |
|
||||
|lastIndexedTxIdBeforeHoles|MetadataTracker|N.A.| |
|
||||
|lastIndexedChangeSetIdBeforeHoles|AclTracker| N.A.| |
|
||||
|running|ALL|ALL||
|
||||
|checkedFirstTransactionTime|MetadataTracker|ACLTracker, MetadataTracker| |
|
||||
|checkedFirstAclTransactionTime|ACLTracker|ACLTracker| |
|
||||
|checkedLastAclTransactionTime|ACLTracker|ACLTracker| |
|
||||
|checkedLastTransactionTime|MetadataTracker|MetadataTracker| |
|
||||
|check|AclTrackerTest|AbstractTracker, CoreAdminHandler::actionCHECK| It seems used only for ACL test purposes, but it is set on the superclass on all trackers|
|
||||
|trackerCycles|ALL |ALL | |
|
||||
|timeToStopIndexing|AclTracker, MetadataTracker|MetadataTrackerTest, SolrInformationServer::getTrackerInitialState |Since it is set in the initial tracker state, it is done for all trackers, but actually it is never set again so the MetadataTracker and AclTracker always use the initial value. |
|
||||
|lastGoodChangeSetCommitTimeInIndex|AclTracker|AclTracker, SolrInformationServer::getTrackerInitialState, SolrInformatonServer::continueState|SolrInformatioServer::continueState is called on every tracker. However, only the AclTracker is using this information. |
|
||||
|lastGoodTxCommitTimeInIndex|MetadataTracker|MetadataTracker, SolrInformationServer::getTrackerInitialState, SolrInformatonServer::continueState| See above|
|
||||
|timeBeforeWhichThereCanBeNoHoles|N.A.| SolrInformationServer::getTrackerInitialState, SolrInformatonServer::continueState | This is set for all trackers but never read.|
|
||||
|lastStartTime| ALL|ALL ||
|
||||
|
||||
_* as part of the MNT-20179, which is being implemented at time of writing, the MetadataTracker won't access anymore to this attribute_
|
||||
|
||||
From the table above we can see there are some intersections between attributes and trackers; specifically we have three different scenarios:
|
||||
|
||||
- n writers, 1 reader (e.g. lastGoodChangeSetCommitTimeInIndex)
|
||||
- n readers, 1 writer (e.g. lastIndexedChangeSetCommitTime, lastIndexedChangeSetId)
|
||||
- n readers, n writers (timeToStopIndexing)
|
||||
|
||||
These scenarios suggest the "unified" approach used within the _TrackerState_ could be improved. Specifically:
|
||||
|
||||
- n writers, 1 reader: dangerous, we need to clarify if the usage of the same property is disjoint or the two writers are supposed to insist on the same property instance/value.
|
||||
- n readers, 1 writer: this could be ok, because only one tracker is mutating the property value.
|
||||
- n readers, n writers: this is the worst scenario; however, it seems only one property (timeToStopIndexing) falls here so the context should be easy to understand and address.
|
||||
|
||||
### Thoughts about the current implementation
|
||||
The current implementation of the _TrackerState_ contains a "mix" of information that can be roughly classified in two categories:
|
||||
|
||||
- things that below to the "stateful" context of the tracker lifecycle (e.g. running, checked, trackerCycles): as a side note, in a refactoring context we need to consider that each tracker instance has also some attribute which belongs to this category but it's not part of the _TrackerState_ (e.g. rollback)
|
||||
- things we want to remember across the tracker cycles (e.g. lastIndexedTxId, checkedFirstTransactionTime)
|
||||
|
||||
The first category is tied to a tracker instance and therefore can be considered part, strictly speaking, of the tracker state.
|
||||
The second category instead, seems to be something shared across the trackers; that is, a place where each tracker (actually only the _MetadataTracker_ and the _AclTracker_ plays a role in this context) can record some information during a tracking cycle.
|
||||
|
||||
The _TrackerState_ creation and initialisation is done in _SolrInformationServer_ (a singleton within each _SolrCore_); at the end of the subsystem initialisation, each tracker has an its own _TrackerState_ instance. Looking at the two points above,
|
||||
that dedicated instance sounds reasonable for the first category (the stateful context of a tracker) while it doesn't make so much sense for the second one. This because each tracker owns a private, decoupled instance of the tracker state but:
|
||||
|
||||
- in the best case it will use only a disjoint and partial portion of that class
|
||||
- in the worst case it will never use that
|
||||
|
||||
As consequence of that, a proper design should divide those two responsibilities, by providing a private copy of the state and a shared instance composed by things we need to remember across tracker cycles.
|
||||
This approach is the main driver of the proposal described in the next section.
|
||||
|
||||
### Proposal
|
||||
As explained above, the proposal would divide the current tracker state in two separate set of attributes.
|
||||
|
||||
#### Tracker State
|
||||
This set includes things that are strictly related with the current state of a tracker (e.g. running, isInRollback, isInMaintenance). These properties are specific for each tracker, they could be expressed in two ways:
|
||||
|
||||
- as instance properties of each tracker (in _AbstractTracker_ or in the concete tracker class)
|
||||
- using a State Pattern [1], in order to formalize the behaviour that a given tracker should have when it is in a specific state.
|
||||
|
||||
Using the first approach, variables like "running", "check", "trackerCycles" should be moved and managed directly or indirectly (_AbstractTracker_) in the tracker class. Each tracker instance must own a private copy of that variable because it can differ between trackers (e.g. _MetadataTracker_ could be running while _AclTracker_ couldn't)
|
||||
|
||||
```Java
|
||||
|
||||
public abstract class AbstractTracker {
|
||||
|
||||
private boolean running;
|
||||
|
||||
...
|
||||
|
||||
|
||||
public boolean isRunning() {
|
||||
return running;
|
||||
}
|
||||
|
||||
public void doTrack() {
|
||||
|
||||
if (isRunning) {
|
||||
// Do something
|
||||
} else {
|
||||
// Dom something else
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
The second approach would transform each tracker in a Finite State Machine (FSM) [2]. Since at the moment the different behaviour is spread across the code with a relevant amount of conditional statements, this approach requires more analysis and implementation effort.
|
||||
The concrete Tracker (or the _AbstractTracker_) would act as a controller which would orchestrate and move between available states, in order to execute the tracking logic.
|
||||
|
||||
#### Tracker Memory/Memento
|
||||
This set of attributes includes all things we need to remember across tracker cycles. The term "Memento" is taken from the popular GoF Design Pattern [3]
|
||||
but in this context the usage is slightly different because here we aren't interested in retain a captured state for doing things like undo/redo operations: the purpose is to persist a set of information between one tracking cycle and another.
|
||||
|
||||
Under this perspective, the Memento/Memory should act as a singleton across all trackers: that is, there should be only one instance for each _SolrCore_. This class will contain all information copied from the _TrackerState_ related with ACLs and Transactions:
|
||||
|
||||
```
|
||||
public class TrackersMemento {
|
||||
|
||||
// ACLs and Transactions are two private inner classes used just for classifying things within
|
||||
// this class. They are not exposed and the caller sees only the TrackersMemory public interface (getters and
|
||||
// setters)
|
||||
|
||||
private class ACLs {
|
||||
private long lastChangeSetIdOnServer;
|
||||
private long lastChangeSetCommitTimeOnServer;
|
||||
private long lastIndexedChangeSetId;
|
||||
private long lastIndexedChangeSetCommitTime;
|
||||
private long lastIndexedChangeSetIdBeforeHoles = -1;
|
||||
|
||||
private volatile boolean checkedFirstAclTransactionTime;
|
||||
private volatile boolean checkedLastAclTransactionTime;
|
||||
|
||||
private long lastGoodChangeSetCommitTimeInIndex;
|
||||
}
|
||||
|
||||
private class Transactions {
|
||||
private long lastIndexedTxCommitTime;
|
||||
private long lastIndexedTxId;
|
||||
private long lastTxCommitTimeOnServer;
|
||||
private long lastTxIdOnServer;
|
||||
private long lastIndexedTxIdBeforeHoles = -1;
|
||||
|
||||
private boolean checkedFirstTransactionTime;
|
||||
|
||||
private boolean checkedLastTransactionTime;
|
||||
|
||||
private long lastGoodTxCommitTimeInIndex;
|
||||
}
|
||||
|
||||
private final ACLs acls = new ACLs();
|
||||
private final Transactions transactions = new Transactions();
|
||||
|
||||
// getter and setter at hosting instance level
|
||||
|
||||
```
|
||||
|
||||
With this class, the tracker subsystem will expose a centralized view of things that need to be recorded. This is the first important difference:
|
||||
in the current implementation each tracker has a copy of (for example) "lastIndexedTxId" regardless if it uses it or not.
|
||||
|
||||
Once we got the centralization, a proper design would enforce/formalize the read/write access to that properties. The purpose is to communicate in a formal way
|
||||
|
||||
- who is allowed to write what
|
||||
- who is allowed to read what
|
||||
|
||||
So, looking at the table at the beginning of this document, after removing things belonging to the tracker state (section above),
|
||||
we can group the remaining properties using two categories (ACLs and Transactions) which can be furtherly split from the access perspective (Read and/or Write).
|
||||
|
||||
So the idea is to create four interfaces, all implemented by the _TrackersMemento_ which would enable
|
||||
|
||||
- ACL properties read access (e.g. _ReadableOnlyAclsMemento_) which will contains all getters related with the _TrackersMemento::ACLs_ properties
|
||||
- ACL properties write (and read) access (e.g. _AclsMemento_ extends _ReadableOnlyAclsMemento_) which will contains all setters related with the _TrackersMemento::ACLs_ properties
|
||||
- Transaction properties read access (e.g. _ReadableOnlyTransactionsMemento_) which will contains all getters related with the _TrackersMemento::Transations_ properties
|
||||
- Transaction properties write (and read) access (e.g. _TransactionsMemento_ extends _ReadableOnlyTransactionsMemento_) which will contains all setters related with the _TrackersMemento::Transactions_ properties
|
||||
|
||||
The _TrackersMemento_ would implement all four interfaces:
|
||||
|
||||
```
|
||||
|
||||
// ReadableOnly intefaces are automatically inherited
|
||||
|
||||
public class TrackersMemento implements AclsMemento, TransactionsMemento {
|
||||
```
|
||||
|
||||
|
||||
|
||||
Using the class hierarchy above, we could improve the formalisation of the different kind of access each tracker would require. Specifically:
|
||||
|
||||
- the memento instance could be available or not in each tracker. That probably means it doesn't make sense to have it available at the _AbstractTracker_ level.
|
||||
- a tracker or a component, depending on its requirements and behaviour, could have a reference to a readable and/or writable (ACLs and/or Transactions) memento.
|
||||
|
||||
For example:
|
||||
|
||||
- _ModelTracker_ won't have any reference because it doesn't need it
|
||||
- _ACLTracker_ would see an _AclsMemento_ with full RW access but a Read-Only view of the _TransactionsMemento_
|
||||
- _CoreAdminHandler_ would have a Read-Only view of both ACLs and Transations memento
|
||||
- _SolrInformationServer_ would a a full access to the _TrackersMemento_, because it create and initialise it
|
||||
|
||||
### Single or Multi Core
|
||||
Being a Singleton, the _TrackerMemento_ could be also designed for storing/serving data belonging to more than one core.
|
||||
This could be useful if the same Solr instance hosts multiple core (e.g. alfresco, archive). In that case the access interfaces and the access pattern would be slightly different
|
||||
because it should be implemented for taking in account also the core name as input parameter.
|
||||
|
||||
The resulting implementation should provide a 1-n cardinality relationship (by means of a dictionary, for example) between the memento instance and the internal stored sets of attributes belonging to different cores.
|
||||
|
||||
***
|
||||
[1] [State Pattern](https://en.wikipedia.org/wiki/State_pattern)
|
||||
[2] [Finite State Machine](https://en.wikipedia.org/wiki/Finite-state_machine)
|
||||
[3] [Memento](https://en.wikipedia.org/wiki/Memento_pattern)
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 304 KiB |
@@ -114,7 +114,7 @@
|
||||
<dependency>
|
||||
<groupId>com.carrotsearch.randomizedtesting</groupId>
|
||||
<artifactId>randomizedtesting-runner</artifactId>
|
||||
<version>2.7.2</version>
|
||||
<version>2.7.3</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
|
||||
+16
-2
@@ -145,6 +145,7 @@ import org.apache.lucene.util.BytesRefBuilder;
|
||||
import org.apache.solr.common.SolrDocument;
|
||||
import org.apache.solr.common.SolrDocumentList;
|
||||
import org.apache.solr.common.SolrInputDocument;
|
||||
import org.apache.solr.common.SolrInputField;
|
||||
import org.apache.solr.common.params.CommonParams;
|
||||
import org.apache.solr.common.params.FacetParams;
|
||||
import org.apache.solr.common.params.ModifiableSolrParams;
|
||||
@@ -1783,6 +1784,7 @@ public class SolrInformationServer implements InformationServer
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@Override
|
||||
public void indexNodes(List<Node> nodes, boolean overwrite, boolean cascade) throws IOException, JSONException
|
||||
{
|
||||
@@ -1890,6 +1892,7 @@ public class SolrInformationServer implements InformationServer
|
||||
continue;
|
||||
}
|
||||
|
||||
|
||||
AddUpdateCommand addDocCmd = new AddUpdateCommand(request);
|
||||
addDocCmd.overwrite = overwrite;
|
||||
|
||||
@@ -2352,6 +2355,10 @@ public class SolrInformationServer implements InformationServer
|
||||
|
||||
if (cachedDoc != null)
|
||||
{
|
||||
ofNullable(cachedDoc.getField("MINHASH"))
|
||||
.map(SolrInputField::getValue)
|
||||
.ifPresent(minHash -> newDoc.setField("MINHASH", minHash));
|
||||
|
||||
// Builds up the new solr doc from the cached content regardless of whether or not it is current
|
||||
List<FieldInstance> fields = AlfrescoSolrDataModel.getInstance().getIndexedFieldNamesForProperty(
|
||||
propertyQName).getFields();
|
||||
@@ -2654,9 +2661,16 @@ public class SolrInformationServer implements InformationServer
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Index information of a node that does not belong to the current shard.
|
||||
* These information are necessary for cascade tracker to work properly.
|
||||
* The information stored are:
|
||||
* nodeDocumentId, cascadeTx
|
||||
*/
|
||||
private void indexNonShardCascade(NodeMetaData nodeMetaData) throws IOException
|
||||
{
|
||||
canUpdate();
|
||||
|
||||
UpdateRequestProcessor processor = null;
|
||||
try (SolrQueryRequest request = newSolrQueryRequest())
|
||||
{
|
||||
@@ -2668,7 +2682,7 @@ public class SolrInformationServer implements InformationServer
|
||||
SolrInputDocument input = new SolrInputDocument();
|
||||
input.addField(FIELD_SOLR4_ID, AlfrescoSolrDataModel.getNodeDocumentId(nodeMetaData.getTenantDomain(), nodeMetaData.getAclId(), nodeMetaData.getId()));
|
||||
input.addField(FIELD_VERSION, 0);
|
||||
input.addField(fieldInstance.getField(), stringPropertyValue.toString());
|
||||
input.addField(fieldInstance.getField(), stringPropertyValue.getValue());
|
||||
cmd.solrDoc = input;
|
||||
processor.processAdd(cmd);
|
||||
|
||||
|
||||
+32
-10
@@ -37,6 +37,7 @@ import org.alfresco.repo.search.impl.QueryParserUtils;
|
||||
import org.alfresco.service.cmr.dictionary.DictionaryService;
|
||||
import org.alfresco.service.cmr.dictionary.PropertyDefinition;
|
||||
import org.alfresco.service.namespace.QName;
|
||||
import org.alfresco.solr.AlfrescoCoreAdminHandler;
|
||||
import org.alfresco.solr.AlfrescoSolrDataModel;
|
||||
import org.alfresco.solr.BoundedDeque;
|
||||
import org.alfresco.solr.InformationServer;
|
||||
@@ -54,6 +55,8 @@ import org.json.JSONException;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import static java.util.Optional.of;
|
||||
|
||||
/*
|
||||
* This tracks two things: transactions and metadata nodes
|
||||
* @author Ahmed Owianå
|
||||
@@ -192,19 +195,39 @@ public class MetadataTracker extends AbstractTracker implements Tracker
|
||||
}
|
||||
|
||||
/**
|
||||
* @return
|
||||
* The {@link ShardState}, as the name suggests, encapsulates/stores the state of the shard which hosts this
|
||||
* {@link MetadataTracker} instance.
|
||||
*
|
||||
* The {@link ShardState} is primarily used in two places:
|
||||
*
|
||||
* <ul>
|
||||
* <li>Transaction tracking: (see {@link #trackTransactions()}): for pulling/tracking transactions from Alfresco</li>
|
||||
* <li>
|
||||
* DynamicSharding: when the {@link MetadataTracker} is running on a slave instance it doesn't actually act
|
||||
* as a tracker, it calls Alfresco to register the state of the node (the shard) without pulling any transactions.
|
||||
* As consequence of that, Alfresco will be aware about the shard which will be included in subsequent queries.
|
||||
* </li>
|
||||
* </ul>
|
||||
*
|
||||
* @return the {@link ShardState} instance which stores the current state of the hosting shard.
|
||||
*/
|
||||
private ShardState getShardState()
|
||||
ShardState getShardState()
|
||||
{
|
||||
TrackerState state = super.getTrackerState();
|
||||
|
||||
ShardState shardstate = ShardStateBuilder.shardState()
|
||||
TrackerState transactionsTrackerState = super.getTrackerState();
|
||||
TrackerState changeSetsTrackerState =
|
||||
of(infoSrv.getAdminHandler())
|
||||
.map(AlfrescoCoreAdminHandler::getTrackerRegistry)
|
||||
.map(registry -> registry.getTrackerForCore(coreName, AclTracker.class))
|
||||
.map(Tracker::getTrackerState)
|
||||
.orElse(transactionsTrackerState);
|
||||
|
||||
return ShardStateBuilder.shardState()
|
||||
.withMaster(isMaster)
|
||||
.withLastUpdated(System.currentTimeMillis())
|
||||
.withLastIndexedChangeSetCommitTime(state.getLastIndexedChangeSetCommitTime())
|
||||
.withLastIndexedChangeSetId(state.getLastIndexedChangeSetId())
|
||||
.withLastIndexedTxCommitTime(state.getLastIndexedTxCommitTime())
|
||||
.withLastIndexedTxId(state.getLastIndexedTxId())
|
||||
.withLastIndexedChangeSetCommitTime(changeSetsTrackerState.getLastIndexedChangeSetCommitTime())
|
||||
.withLastIndexedChangeSetId(changeSetsTrackerState.getLastIndexedChangeSetId())
|
||||
.withLastIndexedTxCommitTime(transactionsTrackerState.getLastIndexedTxCommitTime())
|
||||
.withLastIndexedTxId(transactionsTrackerState.getLastIndexedTxId())
|
||||
.withShardInstance()
|
||||
.withBaseUrl(infoSrv.getBaseUrl())
|
||||
.withPort(infoSrv.getPort())
|
||||
@@ -221,7 +244,6 @@ public class MetadataTracker extends AbstractTracker implements Tracker
|
||||
.endShard()
|
||||
.endShardInstance()
|
||||
.build();
|
||||
return shardstate;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
/*
|
||||
* Copyright (C) 2005-2017 Alfresco Software Limited.
|
||||
*
|
||||
* This file is part of Alfresco
|
||||
*
|
||||
* Alfresco is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU Lesser General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* Alfresco is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General Public License
|
||||
* along with Alfresco. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
package org.alfresco.dataload;
|
||||
|
||||
import org.apache.solr.client.solrj.impl.HttpSolrClient;
|
||||
import org.apache.solr.client.solrj.request.UpdateRequest;
|
||||
import org.apache.solr.common.SolrInputDocument;
|
||||
|
||||
import java.time.LocalDateTime;
|
||||
import java.time.ZoneOffset;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import java.util.Random;
|
||||
|
||||
public class Loader {
|
||||
|
||||
public static void main(String args[]) throws Exception {
|
||||
int num = Integer.parseInt(args[0]);
|
||||
int start = Integer.parseInt(args[1]);
|
||||
//String url = "http://localhost:8985/solr/joel";
|
||||
String url = "http://localhost:8983/solr/collection1";
|
||||
|
||||
HttpSolrClient client = new HttpSolrClient(url);
|
||||
UpdateRequest request = new UpdateRequest();
|
||||
int i = start;
|
||||
LocalDateTime localDate = LocalDateTime.now();
|
||||
|
||||
Random rand = new Random();
|
||||
for(i=start; i<num+start; i++) {
|
||||
String s = rand.nextInt(1000)+"helloworld123";
|
||||
SolrInputDocument doc = new SolrInputDocument();
|
||||
doc.addField("id", i);
|
||||
doc.addField("test_s", s);
|
||||
doc.addField("test_t", "hello world we love you");
|
||||
int year = rand.nextInt(50);
|
||||
int month = rand.nextInt(12);
|
||||
int day = rand.nextInt(30);
|
||||
float f = rand.nextFloat();
|
||||
|
||||
doc.addField("year_i", Integer.toString(year));
|
||||
doc.addField("month_i", Integer.toString(month));
|
||||
doc.addField("day_i", Integer.toString(day));
|
||||
doc.addField("price_f", Float.toString(f));
|
||||
|
||||
LocalDateTime randomDate = localDate.plusDays(rand.nextInt(1000));
|
||||
doc.addField("date_dt", DateTimeFormatter.ISO_INSTANT.format(randomDate.toInstant(ZoneOffset.UTC)));
|
||||
doc.addField("epoch_l", randomDate.atZone(ZoneOffset.UTC).toInstant().toEpochMilli());
|
||||
|
||||
request.add(doc);
|
||||
if(i % 5000 == 0) {
|
||||
request.process(client);
|
||||
client.commit();
|
||||
request = new UpdateRequest();
|
||||
}
|
||||
|
||||
|
||||
for(int l=0; l<5; l++) {
|
||||
String ps = "product"+rand.nextInt(35);
|
||||
doc.addField("prod_ss",ps);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
if((i % 5000) != 0) {
|
||||
request.process(client);
|
||||
client.commit();
|
||||
}
|
||||
|
||||
client.close();
|
||||
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
+19
-7
@@ -227,24 +227,34 @@ public abstract class AbstractAlfrescoDistributedTest extends SolrTestInitialize
|
||||
* @throws Exception
|
||||
*/
|
||||
public static void waitForDocCount(Query query, int count, long waitMillis) throws Exception
|
||||
{
|
||||
waitForDocCount(luceneToSolrQuery(query), count, waitMillis);
|
||||
}
|
||||
|
||||
public static void waitForDocCount(ModifiableSolrParams query, int count, long waitMillis) throws Exception
|
||||
{
|
||||
long begin = System.currentTimeMillis();
|
||||
//TODO: Support multiple cores per jetty
|
||||
SolrClient standaloneClient = getStandaloneClients().get(0); //Get the first one
|
||||
waitForDocCountCore(standaloneClient, luceneToSolrQuery(query), count, waitMillis, begin);
|
||||
waitForDocCountCore(standaloneClient, query, count, waitMillis, begin);
|
||||
waitForShardsCount(query, count, waitMillis, begin);
|
||||
}
|
||||
|
||||
public static void assertShardCount(int shardNumber, Query query, int count) throws Exception {
|
||||
public static void assertShardCount(int shardNumber, Query query, int count) throws Exception
|
||||
{
|
||||
assertShardCount(shardNumber, luceneToSolrQuery(query), count);
|
||||
}
|
||||
|
||||
public static void assertShardCount(int shardNumber, ModifiableSolrParams query, int count) throws Exception
|
||||
{
|
||||
List<SolrClient> clients = getShardedClients();
|
||||
SolrClient client = clients.get(shardNumber);
|
||||
|
||||
QueryResponse response = client.query(luceneToSolrQuery(query));
|
||||
QueryResponse response = client.query(query);
|
||||
int totalHits = (int) response.getResults().getNumFound();
|
||||
if (count != totalHits) {
|
||||
throw new Exception("Expecting " + count + " docs on shard " + shardNumber + " , found " + totalHits);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
private static String escapeQueryClause(String query)
|
||||
@@ -291,15 +301,16 @@ public abstract class AbstractAlfrescoDistributedTest extends SolrTestInitialize
|
||||
public static void waitForShardsCount(Query query, int count, long waitMillis, long start) throws Exception
|
||||
{
|
||||
SolrQuery solrQuery = luceneToSolrQuery(query);
|
||||
solrQuery.setParam("shards", shards);
|
||||
waitForShardsCount(solrQuery, count, waitMillis, start);
|
||||
}
|
||||
|
||||
|
||||
public static void waitForShardsCount(SolrQuery query, int count, long waitMillis, long start) throws Exception
|
||||
public static void waitForShardsCount(ModifiableSolrParams query, int count, long waitMillis, long start) throws Exception
|
||||
{
|
||||
|
||||
long timeOut = start+waitMillis;
|
||||
int totalCount = 0;
|
||||
query.set("shards", shards);
|
||||
SolrClient clientShard = clientShards.get(0);
|
||||
while (System.currentTimeMillis() < timeOut)
|
||||
{
|
||||
@@ -450,7 +461,7 @@ public abstract class AbstractAlfrescoDistributedTest extends SolrTestInitialize
|
||||
}
|
||||
|
||||
public static void waitForDocCountCore(SolrClient client,
|
||||
SolrQuery query,
|
||||
ModifiableSolrParams query,
|
||||
long expectedNumFound,
|
||||
long waitMillis,
|
||||
long startMillis)
|
||||
@@ -733,6 +744,7 @@ public abstract class AbstractAlfrescoDistributedTest extends SolrTestInitialize
|
||||
request.setMethod(SolrRequest.METHOD.POST);
|
||||
return request;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Returns the QueryResponse from {@link #queryRandomShard}
|
||||
|
||||
+42
-2
@@ -18,6 +18,8 @@
|
||||
*/
|
||||
package org.alfresco.solr;
|
||||
|
||||
import static java.util.Optional.of;
|
||||
import static java.util.Optional.ofNullable;
|
||||
import static junit.framework.TestCase.assertNotNull;
|
||||
import static junit.framework.TestCase.assertTrue;
|
||||
import static org.alfresco.repo.search.adaptor.lucene.QueryConstants.FIELD_ACLID;
|
||||
@@ -68,6 +70,7 @@ import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
|
||||
import org.alfresco.model.ContentModel;
|
||||
import org.alfresco.repo.index.shard.ShardState;
|
||||
import org.alfresco.repo.tenant.TenantService;
|
||||
import org.alfresco.service.cmr.repository.ChildAssociationRef;
|
||||
import org.alfresco.service.cmr.repository.NodeRef;
|
||||
@@ -86,11 +89,11 @@ import org.alfresco.solr.client.StringPropertyValue;
|
||||
import org.alfresco.solr.client.Transaction;
|
||||
import org.alfresco.util.ISO9075;
|
||||
import org.alfresco.util.Pair;
|
||||
import org.apache.solr.SolrTestCaseJ4;
|
||||
import org.apache.solr.SolrTestCaseJ4.XmlDoc;
|
||||
import org.apache.solr.common.SolrInputDocument;
|
||||
import org.apache.solr.common.params.CoreAdminParams;
|
||||
import org.apache.solr.common.params.ModifiableSolrParams;
|
||||
import org.apache.solr.common.params.SolrParams;
|
||||
import org.apache.solr.common.util.NamedList;
|
||||
import org.apache.solr.common.util.XML;
|
||||
import org.apache.solr.core.CoreContainer;
|
||||
@@ -108,6 +111,8 @@ import org.apache.solr.update.CommitUpdateCommand;
|
||||
public class AlfrescoSolrUtils
|
||||
{
|
||||
public static final String TEST_NAMESPACE = "http://www.alfresco.org/test/solrtest";
|
||||
public static long MAX_WAIT_TIME = 80000;
|
||||
|
||||
/**
|
||||
* Get transaction.
|
||||
* @param deletes
|
||||
@@ -884,5 +889,40 @@ public class AlfrescoSolrUtils
|
||||
assertTrue("There must be a searcher for "+coreName, ((Integer)coreSummary.get("Number of Searchers")) > 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Asserts that the input {@link ShardState} and the CoreAdmin.SUMMARY response give the same information.
|
||||
*
|
||||
* @param state the {@link ShardState} instance.
|
||||
* @param core the target {@link SolrCore} instance.
|
||||
*/
|
||||
public static void assertShardAndCoreSummaryConsistency(ShardState state, SolrCore core) {
|
||||
SolrParams params =
|
||||
new ModifiableSolrParams()
|
||||
.add(CoreAdminParams.CORE, core.getName())
|
||||
.add(CoreAdminParams.ACTION, "SUMMARY");
|
||||
|
||||
}
|
||||
SolrQueryRequest request = new LocalSolrQueryRequest(core, params);
|
||||
SolrQueryResponse response = new SolrQueryResponse();
|
||||
coreAdminHandler(core).handleRequest(request, response);
|
||||
|
||||
NamedList<?> summary =
|
||||
ofNullable(response.getValues())
|
||||
.map(values -> values.get("Summary"))
|
||||
.map(NamedList.class::cast)
|
||||
.map(values -> values.get(core.getName()))
|
||||
.map(NamedList.class::cast)
|
||||
.orElseGet(NamedList::new);
|
||||
|
||||
assertEquals(state.getLastIndexedChangeSetId(), summary.get("Id for last Change Set in index"));
|
||||
assertEquals(state.getLastIndexedChangeSetCommitTime(), summary.get("Last Index Change Set Commit Time"));
|
||||
assertEquals(state.getLastIndexedTxCommitTime(), summary.get("Last Index TX Commit Time"));
|
||||
assertEquals(state.getLastIndexedTxId(), summary.get("Id for last TX in index"));
|
||||
}
|
||||
|
||||
public static AlfrescoCoreAdminHandler coreAdminHandler(SolrCore core) {
|
||||
return of(core).map(SolrCore::getCoreContainer)
|
||||
.map(CoreContainer::getMultiCoreHandler)
|
||||
.map(AlfrescoCoreAdminHandler.class::cast)
|
||||
.orElseThrow(() -> new IllegalStateException("Cannot retrieve the Core Admin Handler on this test core."));
|
||||
}
|
||||
}
|
||||
+20
-1
@@ -16,10 +16,11 @@
|
||||
* You should have received a copy of the GNU Lesser General Public License
|
||||
* along with Alfresco. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
package org.alfresco.solr.query.afts;
|
||||
package org.alfresco.solr.dataload;
|
||||
|
||||
import static java.util.Arrays.asList;
|
||||
import static java.util.Collections.singletonList;
|
||||
import static java.util.stream.IntStream.range;
|
||||
import static org.alfresco.model.ContentModel.ASSOC_CONTAINS;
|
||||
import static org.alfresco.model.ContentModel.ASSOC_CHILDREN;
|
||||
import static org.alfresco.model.ContentModel.PROP_NAME;
|
||||
@@ -28,6 +29,8 @@ import static org.alfresco.service.namespace.NamespaceService.CONTENT_MODEL_1_0_
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.addNode;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.createGUID;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.addStoreRoot;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getNode;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getNodeMetaData;
|
||||
|
||||
import org.alfresco.model.ContentModel;
|
||||
import org.alfresco.service.cmr.repository.ChildAssociationRef;
|
||||
@@ -39,19 +42,27 @@ import org.alfresco.service.cmr.repository.datatype.Duration;
|
||||
import org.alfresco.service.namespace.QName;
|
||||
import org.alfresco.solr.AlfrescoSolrConstants;
|
||||
import org.alfresco.solr.AlfrescoSolrDataModel;
|
||||
import org.alfresco.solr.client.Acl;
|
||||
import org.alfresco.solr.client.ContentPropertyValue;
|
||||
import org.alfresco.solr.client.MLTextPropertyValue;
|
||||
import org.alfresco.solr.client.MultiPropertyValue;
|
||||
import org.alfresco.solr.client.Node;
|
||||
import org.alfresco.solr.client.NodeMetaData;
|
||||
import org.alfresco.solr.client.PropertyValue;
|
||||
import org.alfresco.solr.client.StringPropertyValue;
|
||||
import org.alfresco.solr.client.Transaction;
|
||||
import org.alfresco.util.ISO9075;
|
||||
import org.apache.solr.client.solrj.io.Tuple;
|
||||
import org.apache.solr.core.SolrCore;
|
||||
import org.apache.solr.util.TestHarness;
|
||||
|
||||
import java.util.AbstractMap;
|
||||
import java.util.Date;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
/**
|
||||
* Test datasets provider.
|
||||
@@ -855,6 +866,14 @@ public class TestDataProvider implements AlfrescoSolrConstants
|
||||
return testProperties;
|
||||
}
|
||||
|
||||
public static Map.Entry<List<Node>, List<NodeMetaData>> nSampleNodesWithSampleContent(Acl acl, Transaction txn, int howManyNodes) {
|
||||
|
||||
List<Node> nodes = range(0, howManyNodes).mapToObj(index -> getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED)).collect(Collectors.toList());
|
||||
List<NodeMetaData> metadata = nodes.stream().map(node -> getNodeMetaData(node, txn, acl, "mike", null, false)).collect(Collectors.toList());
|
||||
|
||||
return new AbstractMap.SimpleImmutableEntry<>(nodes, metadata);
|
||||
}
|
||||
|
||||
public NodeRef newNodeRef()
|
||||
{
|
||||
return new NodeRef(new StoreRef("workspace", "SpacesStore"), createGUID());
|
||||
+268
-222
@@ -19,50 +19,75 @@
|
||||
|
||||
package org.alfresco.solr.query;
|
||||
|
||||
import org.alfresco.model.ContentModel;
|
||||
import org.alfresco.repo.search.adaptor.lucene.QueryConstants;
|
||||
import org.alfresco.solr.AbstractAlfrescoSolrTests;
|
||||
import org.alfresco.solr.client.*;
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import org.alfresco.solr.client.Acl;
|
||||
import org.alfresco.solr.client.AclChangeSet;
|
||||
import org.alfresco.solr.client.AclReaders;
|
||||
import org.alfresco.solr.client.Node;
|
||||
import org.alfresco.solr.client.NodeMetaData;
|
||||
import org.alfresco.solr.client.SOLRAPIQueueClient;
|
||||
import org.alfresco.solr.client.StringPropertyValue;
|
||||
import org.alfresco.solr.client.Transaction;
|
||||
import org.apache.lucene.index.Term;
|
||||
import org.apache.lucene.search.BooleanClause;
|
||||
import org.apache.lucene.search.BooleanQuery;
|
||||
import org.apache.lucene.search.LegacyNumericRangeQuery;
|
||||
import org.apache.lucene.search.TermQuery;
|
||||
import org.apache.lucene.util.LuceneTestCase;
|
||||
import org.apache.solr.common.params.ModifiableSolrParams;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Random;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import static java.util.Arrays.asList;
|
||||
import static java.util.Arrays.stream;
|
||||
import static java.util.Collections.singletonList;
|
||||
import static java.util.stream.IntStream.range;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.*;
|
||||
|
||||
@LuceneTestCase.SuppressCodecs({"Appending","Lucene3x","Lucene40","Lucene41","Lucene42","Lucene43", "Lucene44", "Lucene45","Lucene46","Lucene47","Lucene48","Lucene49"})
|
||||
public class AlfrescoSolrFingerprintTest extends AbstractAlfrescoSolrTests
|
||||
{
|
||||
private static Log logger = LogFactory.getLog(AlfrescoSolrFingerprintTest.class);
|
||||
private static long MAX_WAIT_TIME = 80000;
|
||||
|
||||
@BeforeClass
|
||||
public static void beforeClass() throws Exception
|
||||
{
|
||||
initAlfrescoCore("schema-fingerprint.xml");
|
||||
initAlfrescoCore("schema.xml");
|
||||
}
|
||||
|
||||
private Acl acl;
|
||||
|
||||
@Before
|
||||
public void setUp() throws Exception {
|
||||
// if you override setUp or tearDown, you better callf
|
||||
// the super classes version
|
||||
//clearIndex();
|
||||
//assertU(commit());
|
||||
public void prepare() throws Exception
|
||||
{
|
||||
AclChangeSet aclChangeSet = getAclChangeSet(1);
|
||||
acl = getAcl(aclChangeSet);
|
||||
Acl acl2 = getAcl(aclChangeSet);
|
||||
|
||||
AclReaders aclReaders = getAclReaders(aclChangeSet, acl, singletonList("joel"), singletonList("phil"), null);
|
||||
AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, singletonList("jim"), singletonList("phil"), null);
|
||||
|
||||
indexAclChangeSet(aclChangeSet,
|
||||
asList(acl, acl2),
|
||||
asList(aclReaders, aclReaders2));
|
||||
|
||||
// Check for the ACL state stamp.
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!ACLTX")), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_ACLTXID, aclChangeSet.getId(), aclChangeSet.getId() + 1, true, false), BooleanClause.Occur.MUST));
|
||||
BooleanQuery waitForQuery = builder.build();
|
||||
waitForDocCount(waitForQuery, 1, MAX_WAIT_TIME);
|
||||
}
|
||||
|
||||
@After
|
||||
public void clearQueue() throws Exception {
|
||||
public void clearQueue()
|
||||
{
|
||||
SOLRAPIQueueClient.nodeMetaDataMap.clear();
|
||||
SOLRAPIQueueClient.transactionQueue.clear();
|
||||
SOLRAPIQueueClient.aclChangeSetQueue.clear();
|
||||
@@ -70,234 +95,255 @@ public class AlfrescoSolrFingerprintTest extends AbstractAlfrescoSolrTests
|
||||
SOLRAPIQueueClient.aclMap.clear();
|
||||
SOLRAPIQueueClient.nodeMap.clear();
|
||||
SOLRAPIQueueClient.nodeContentMap.clear();
|
||||
|
||||
clearIndex();
|
||||
assertU(commit());
|
||||
}
|
||||
|
||||
|
||||
@Test
|
||||
public void testBasciFingerPrint() throws Exception
|
||||
private void makeSureTransactionHasBeenIndexed(long transactionId) throws Exception
|
||||
{
|
||||
/*
|
||||
* Create and index an AclChangeSet.
|
||||
*/
|
||||
|
||||
logger.info("######### Starting fingerprint test ###########");
|
||||
AclChangeSet aclChangeSet = getAclChangeSet(1);
|
||||
|
||||
Acl acl = getAcl(aclChangeSet);
|
||||
Acl acl2 = getAcl(aclChangeSet);
|
||||
|
||||
|
||||
AclReaders aclReaders = getAclReaders(aclChangeSet, acl, list("joel"), list("phil"), null);
|
||||
AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, list("jim"), list("phil"), null);
|
||||
|
||||
|
||||
indexAclChangeSet(aclChangeSet,
|
||||
list(acl, acl2),
|
||||
list(aclReaders, aclReaders2));
|
||||
|
||||
|
||||
//Check for the ACL state stamp.
|
||||
//Check for the TXN state stamp.
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!ACLTX")), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_ACLTXID, aclChangeSet.getId(), aclChangeSet.getId() + 1, true, false), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!TX")), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_TXID, transactionId, transactionId + 1, true, false), BooleanClause.Occur.MUST));
|
||||
BooleanQuery waitForQuery = builder.build();
|
||||
waitForDocCount(waitForQuery, 1, MAX_WAIT_TIME);
|
||||
}
|
||||
|
||||
logger.info("#################### Passed First Test ##############################");
|
||||
|
||||
/*
|
||||
* Create and index a Transaction
|
||||
*/
|
||||
|
||||
//First create a transaction.
|
||||
@Test
|
||||
public void testBasicFingerprint() throws Exception
|
||||
{
|
||||
Transaction txn = getTransaction(0, 4);
|
||||
|
||||
//Next create two nodes to update for the transaction
|
||||
Node node1 = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
Node node2 = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
Node node3 = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
Node node4 = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
|
||||
|
||||
//Next create the NodeMetaData for each node. TODO: Add more metadata
|
||||
NodeMetaData nodeMetaData1 = getNodeMetaData(node1, txn, acl, "mike", null, false);
|
||||
NodeMetaData nodeMetaData2 = getNodeMetaData(node2, txn, acl, "mike", null, false);
|
||||
NodeMetaData nodeMetaData3 = getNodeMetaData(node3, txn, acl, "mike", null, false);
|
||||
NodeMetaData nodeMetaData4 = getNodeMetaData(node4, txn, acl, "mike", null, false);
|
||||
|
||||
List<String> content = new ArrayList();
|
||||
Random randomizer = new Random(1);
|
||||
String aFirstToken = Integer.toString(Math.abs(randomizer.nextInt()));
|
||||
|
||||
indexTransaction(txn,
|
||||
asList(node1, node2, node3, node4),
|
||||
asList(nodeMetaData1, nodeMetaData2, nodeMetaData3, nodeMetaData4),
|
||||
randomTextContent());
|
||||
|
||||
makeSureTransactionHasBeenIndexed(txn.getId());
|
||||
makeSureContentNodesHaveBeenIndexed("mike", aFirstToken, 4);
|
||||
|
||||
assertFingerprintQueryCorrectness(node1.getId(),
|
||||
"*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']");
|
||||
|
||||
assertFingerprintQueryCorrectness(nodeMetaData1.getNodeRef().getId(),
|
||||
"*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']");
|
||||
|
||||
assertFingerprintQueryCorrectness(node1.getId() + "_70",
|
||||
"*[count(//doc)= 2]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']");
|
||||
|
||||
assertFingerprintQueryCorrectness(nodeMetaData1.getNodeRef().getId() + "_70",
|
||||
"*[count(//doc)= 2]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']");
|
||||
|
||||
assertFingerprintQueryCorrectness(node1.getId() + "_45",
|
||||
"*[count(//doc)= 3]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']");
|
||||
|
||||
assertFingerprintQueryCorrectness(nodeMetaData1.getNodeRef().getId() + "_45",
|
||||
"*[count(//doc)= 3]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']");
|
||||
|
||||
assertFingerprintQueryCorrectness(node4.getId() + "_30",
|
||||
"*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']");
|
||||
|
||||
assertFingerprintQueryCorrectness(nodeMetaData4.getNodeRef().getId() + "_30",
|
||||
"*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']");
|
||||
|
||||
assertFingerprintQueryCorrectness(node4.getId(),
|
||||
"*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']");
|
||||
|
||||
assertFingerprintQueryCorrectness(nodeMetaData4.getNodeRef().getId(),
|
||||
"*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']");
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFingerprintStillExistsAfterNodeMetadataUpdate() throws Exception
|
||||
{
|
||||
Transaction txn = getTransaction(0, 1);
|
||||
Node fileNode = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
NodeMetaData fileMetaData = getNodeMetaData(fileNode, txn, acl, "mike", null, false);
|
||||
|
||||
indexTransaction(
|
||||
txn,
|
||||
singletonList(fileNode),
|
||||
singletonList(fileMetaData),
|
||||
singletonList("This is a text content which is longer than the default hello world " + fileNode.getId() +
|
||||
" returned by the Mock SOLRAPIQueueClient. This is needed because the \"min_hash\" field type " +
|
||||
"definition in Solr doesn't take in account fields which produce less than 5 tokens (see the " +
|
||||
"ShingleFilter settings)."));
|
||||
|
||||
makeSureTransactionHasBeenIndexed(txn.getId());
|
||||
makeSureContentNodeHasBeenIndexed(fileNode, "mike", "world");
|
||||
|
||||
assertFingerprintQueryCorrectness(fileNode.getId(), "*[count(//doc)=1]","//result/doc[1]/long[@name='DBID'][.='" + fileNode.getId() + "']");
|
||||
|
||||
// Let's update the test node
|
||||
fileMetaData.setOwner("Andrea");
|
||||
fileMetaData.getProperties().put(ContentModel.PROP_TITLE, new StringPropertyValue("This is the new file \"title\" metadata attribute."));
|
||||
reindexTransactionId(txn.getId());
|
||||
|
||||
makeSureContentNodeHasBeenIndexed(fileNode, "Andrea", "world");
|
||||
|
||||
assertFingerprintQueryCorrectness(fileNode.getId(), "*[count(//doc)=1]","//result/doc[1]/long[@name='DBID'][.='" + fileNode.getId() + "']");
|
||||
}
|
||||
|
||||
/**
|
||||
* Queries the index using a token from the (dummy) text produced by the test framework ("world", actually).
|
||||
* Once the query returns a positive result we are sure the ContentTracker
|
||||
*
|
||||
* <ol>
|
||||
* <li>
|
||||
* Fetched the text content associated with the current node, from Alfresco
|
||||
* </li>
|
||||
* <li>
|
||||
* Computed a fingerprint (using the retrieved text) for the node
|
||||
* </li>
|
||||
* <li>
|
||||
* Updated the node definition in the (Solr)ContentStore and in Solr
|
||||
* </li>
|
||||
* </ol>
|
||||
*
|
||||
* Last but not least, we are also making sure that CommitTracker executed its cycle as well (otherwise document
|
||||
* wouldn't be searchable).
|
||||
*
|
||||
* @param node an addition term which will be appended as a required clause in the executed query.
|
||||
* @param testTerm a term which is supposed to be in the indexed content
|
||||
* @param owner the #FIELD_OWNER which will be used as an additional required query clause.
|
||||
* @throws Exception in case the MAX_WAIT_TIME is reached and the node is not in results.
|
||||
*/
|
||||
private void makeSureContentNodeHasBeenIndexed(final Node node, final String owner, String testTerm) throws Exception
|
||||
{
|
||||
waitForDocCount(new TermQuery(new Term(QueryConstants.FIELD_READER, "jim")), 1, MAX_WAIT_TIME);
|
||||
waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), 1, MAX_WAIT_TIME);
|
||||
|
||||
waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", Long.toString(node.getId()))), 1, MAX_WAIT_TIME);
|
||||
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_OWNER, owner)), BooleanClause.Occur.MUST));
|
||||
waitForDocCount(builder.build(), 1, MAX_WAIT_TIME);
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Queries the index using a token from the (dummy) text produced by the test framework.
|
||||
* Once the query returns a positive result we are sure the ContentTracker
|
||||
*
|
||||
* <ol>
|
||||
* <li>
|
||||
* Fetched the text content associated with the test nodes, from Alfresco
|
||||
* </li>
|
||||
* <li>
|
||||
* Computed a fingerprint (using the retrieved text) for each node
|
||||
* </li>
|
||||
* <li>
|
||||
* Updated the nodes definitions in the (Solr)ContentStore and in Solr
|
||||
* </li>
|
||||
* </ol>
|
||||
*
|
||||
* Last but not least, we are also making sure that CommitTracker executed its cycle as well (otherwise documents
|
||||
* wouldn't be searchable).
|
||||
*
|
||||
* @param owner the #FIELD_OWNER which will be used as an additional required query clause.
|
||||
* @param testTerm a term which is supposed to be in the indexed content
|
||||
* @throws Exception in case the MAX_WAIT_TIME is reached and the node is not in results.
|
||||
*/
|
||||
private void makeSureContentNodesHaveBeenIndexed(final String owner, String testTerm, final int expectedCount) throws Exception
|
||||
{
|
||||
waitForDocCount(new TermQuery(new Term(QueryConstants.FIELD_READER, "jim")), 1, MAX_WAIT_TIME);
|
||||
waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), expectedCount, MAX_WAIT_TIME);
|
||||
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_OWNER, owner)), BooleanClause.Occur.MUST));
|
||||
waitForDocCount(builder.build(), expectedCount, MAX_WAIT_TIME);
|
||||
}
|
||||
|
||||
/**
|
||||
* Asserts that a query in the format <pre>FINGERPRINT:<DBID></pre> correctly returns the node we are testing.
|
||||
*
|
||||
* @param id the node identifier.
|
||||
*/
|
||||
private void assertFingerprintQueryCorrectness(long id, String ... assertions)
|
||||
{
|
||||
assertFingerprintQueryCorrectness(String.valueOf(id), assertions);
|
||||
}
|
||||
|
||||
/**
|
||||
* Asserts that a query in the format <pre>FINGERPRINT:<DBID></pre> correctly returns the node we are testing.
|
||||
*
|
||||
* @param id the node identifier.
|
||||
*/
|
||||
private void assertFingerprintQueryCorrectness(String id, String ... assertions)
|
||||
{
|
||||
ModifiableSolrParams params = new ModifiableSolrParams()
|
||||
.add("q", "FINGERPRINT:" + id)
|
||||
.add("qt", "/afts")
|
||||
.add("start", "0")
|
||||
.add("rows", "6")
|
||||
.add("sort", "id asc");
|
||||
// .add("fq", "{!afts}AUTHORITY_FILTER_FROM_JSON");
|
||||
|
||||
SolrServletRequest req = areq(params, "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [ \"joel\"], \"tenants\": [ \"\" ]}");
|
||||
assertQ(req, assertions);
|
||||
}
|
||||
|
||||
private List<String> randomTextContent()
|
||||
{
|
||||
int[] sizes = {2000, 1000, 1500, 750};
|
||||
|
||||
Random r = new Random(1);
|
||||
String token1 = Integer.toString(Math.abs(r.nextInt()));
|
||||
|
||||
for(int i=0; i<4; i++) {
|
||||
Random rand = new Random(1);
|
||||
StringBuilder buf = new StringBuilder();
|
||||
int size = sizes[i];
|
||||
for(int s=0; s<size; s++) {
|
||||
if(s>0) {
|
||||
buf.append(" ");
|
||||
}
|
||||
buf.append(Integer.toString(Math.abs(rand.nextInt())));
|
||||
}
|
||||
content.add(buf.toString());
|
||||
}
|
||||
|
||||
//Index the transaction, nodes, and nodeMetaDatas.
|
||||
//Note that the content is automatically created by the test framework.
|
||||
indexTransaction(txn,
|
||||
list(node1, node2, node3, node4),
|
||||
list(nodeMetaData1, nodeMetaData2, nodeMetaData3, nodeMetaData4),
|
||||
content);
|
||||
|
||||
//Check for the TXN state stamp.
|
||||
logger.info("#################### Started Second Test ##############################");
|
||||
builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!TX")), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_TXID, txn.getId(), txn.getId() + 1, true, false), BooleanClause.Occur.MUST));
|
||||
waitForQuery = builder.build();
|
||||
|
||||
waitForDocCount(waitForQuery, 1, MAX_WAIT_TIME);
|
||||
logger.info("#################### Passed Second Test ##############################");
|
||||
|
||||
/*
|
||||
* Query the index for the content
|
||||
*/
|
||||
|
||||
waitForDocCount(new TermQuery(new Term(QueryConstants.FIELD_READER, "jim")), 1, MAX_WAIT_TIME);
|
||||
waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", token1)), 4, MAX_WAIT_TIME);
|
||||
|
||||
logger.info("#################### Passed Third Test ##############################");
|
||||
|
||||
ModifiableSolrParams params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + node1.getId()); //Query for an id in the content field. The node id is automatically populated into the content field by test framework
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl", "DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
SolrServletRequest req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)=4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='" + node1.getId() + "']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='" + node3.getId() + "']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='" + node2.getId() + "']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='" + node4.getId() + "']");
|
||||
|
||||
params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + node1.getId() + "_70"); //Query for an id in the content field. The node id is automatically populated into the content field by test framework
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl","DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)= 2]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']");
|
||||
|
||||
params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + node1.getId()+"_45"); //Query for an id in the content field. The node id is automatically populated into the content field by test framework
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl","DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)= 3]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node2.getId()+"']");
|
||||
|
||||
params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + node1.getId()+"_30");
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl","DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']");
|
||||
|
||||
|
||||
params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + node4.getId());
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl","DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node4.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node1.getId()+"']");
|
||||
|
||||
//Test nodeRef
|
||||
|
||||
params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + nodeMetaData1.getNodeRef().getId()); //Query for an id in the content field. The node id is automatically populated into the content field by test framework
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl", "DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
|
||||
req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)=4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='" + node1.getId() + "']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='" + node3.getId() + "']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='" + node2.getId() + "']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='" + node4.getId() + "']");
|
||||
|
||||
params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + nodeMetaData1.getNodeRef().getId() + "_70"); //Query for an id in the content field. The node id is automatically populated into the content field by test framework
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl","DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)= 2]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']");
|
||||
|
||||
params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + nodeMetaData1.getNodeRef().getId()+"_45"); //Query for an id in the content field. The node id is automatically populated into the content field by test framework
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl","DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)= 3]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node2.getId()+"']");
|
||||
|
||||
params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + nodeMetaData1.getNodeRef().getId()+"_30");
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl","DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']");
|
||||
|
||||
|
||||
params = new ModifiableSolrParams();
|
||||
params.add("q", "FINGERPRINT:" + nodeMetaData4.getNodeRef().getId());
|
||||
params.add("qt", "/afts");
|
||||
params.add("fl","DBID,score");
|
||||
params.add("start", "0");
|
||||
params.add("rows", "6");
|
||||
req = areq(params, null);
|
||||
assertQ(req, "*[count(//doc)= 4]",
|
||||
"//result/doc[1]/long[@name='DBID'][.='"+node4.getId()+"']",
|
||||
"//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']",
|
||||
"//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']",
|
||||
"//result/doc[4]/long[@name='DBID'][.='"+node1.getId()+"']");
|
||||
|
||||
return stream(sizes)
|
||||
.mapToObj(item -> {
|
||||
Random randomizer = new Random(1);
|
||||
return range(0, item)
|
||||
.mapToObj(i -> randomizer.nextInt())
|
||||
.map(Object::toString)
|
||||
.collect(Collectors.joining(" "));})
|
||||
.collect(Collectors.toList());
|
||||
}
|
||||
}
|
||||
|
||||
+156
-70
@@ -18,6 +18,7 @@
|
||||
*/
|
||||
package org.alfresco.solr.query;
|
||||
|
||||
import static java.util.Arrays.asList;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAcl;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAclChangeSet;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAclReaders;
|
||||
@@ -25,12 +26,13 @@ import static org.alfresco.solr.AlfrescoSolrUtils.getNode;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getNodeMetaData;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getTransaction;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.indexAclChangeSet;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.list;
|
||||
import static java.util.Collections.singletonList;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Random;
|
||||
|
||||
import org.alfresco.model.ContentModel;
|
||||
import org.alfresco.repo.search.adaptor.lucene.QueryConstants;
|
||||
import org.alfresco.solr.AbstractAlfrescoDistributedTest;
|
||||
import org.alfresco.solr.client.Acl;
|
||||
@@ -38,6 +40,7 @@ import org.alfresco.solr.client.AclChangeSet;
|
||||
import org.alfresco.solr.client.AclReaders;
|
||||
import org.alfresco.solr.client.Node;
|
||||
import org.alfresco.solr.client.NodeMetaData;
|
||||
import org.alfresco.solr.client.StringPropertyValue;
|
||||
import org.alfresco.solr.client.Transaction;
|
||||
import org.apache.lucene.index.Term;
|
||||
import org.apache.lucene.search.BooleanClause;
|
||||
@@ -60,31 +63,26 @@ import org.junit.Test;
|
||||
@LuceneTestCase.SuppressCodecs({"Appending","Lucene3x","Lucene40","Lucene41","Lucene42","Lucene43", "Lucene44", "Lucene45","Lucene46","Lucene47","Lucene48","Lucene49"})
|
||||
public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDistributedTest
|
||||
{
|
||||
private static Node[] nodes = new Node[4];
|
||||
private static NodeMetaData[] nodesMetada = new NodeMetaData[4];
|
||||
|
||||
private static long MAX_WAIT_TIME = 80000;
|
||||
|
||||
private static Node[] NODES = new Node[4];
|
||||
private static NodeMetaData[] NODES_METADATA = new NodeMetaData[4];
|
||||
private static Acl ACL;
|
||||
|
||||
@BeforeClass
|
||||
private static void initData() throws Throwable
|
||||
{
|
||||
initSolrServers(2,"DistributedAlfrescoSolrFingerPrintTest",null);
|
||||
/*
|
||||
* Create and index an AclChangeSet.
|
||||
*/
|
||||
initSolrServers(2,getClassName(),null);
|
||||
|
||||
AclChangeSet aclChangeSet = getAclChangeSet(1);
|
||||
|
||||
Acl acl = getAcl(aclChangeSet);
|
||||
ACL = getAcl(aclChangeSet);
|
||||
Acl acl2 = getAcl(aclChangeSet);
|
||||
|
||||
AclReaders aclReaders = getAclReaders(aclChangeSet, ACL, singletonList("joel"), singletonList("phil"), null);
|
||||
AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, singletonList("jim"), singletonList("phil"), null);
|
||||
|
||||
AclReaders aclReaders = getAclReaders(aclChangeSet, acl, list("joel"), list("phil"), null);
|
||||
AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, list("jim"), list("phil"), null);
|
||||
|
||||
|
||||
indexAclChangeSet(aclChangeSet,
|
||||
list(acl, acl2),
|
||||
list(aclReaders, aclReaders2));
|
||||
|
||||
indexAclChangeSet(aclChangeSet, asList(ACL, acl2), asList(aclReaders, aclReaders2));
|
||||
|
||||
//Check for the ACL state stamp.
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
@@ -93,27 +91,22 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
BooleanQuery waitForQuery = builder.build();
|
||||
waitForDocCountAllCores(waitForQuery, 1, 80000);
|
||||
|
||||
/*
|
||||
* Create and index a Transaction
|
||||
*/
|
||||
|
||||
//First create a transaction.
|
||||
Transaction txn = getTransaction(0, 4);
|
||||
|
||||
//Next create two nodes to update for the transaction
|
||||
nodes[0] = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
nodes[1] = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
nodes[2] = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
nodes[3] = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
//Next create two NODES to update for the transaction
|
||||
NODES[0] = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED);
|
||||
NODES[1] = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED);
|
||||
NODES[2] = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED);
|
||||
NODES[3] = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED);
|
||||
|
||||
|
||||
//Next create the NodeMetaData for each node. TODO: Add more metadata
|
||||
nodesMetada[0] = getNodeMetaData(nodes[0], txn, acl, "mike", null, false);
|
||||
nodesMetada[1] = getNodeMetaData(nodes[1], txn, acl, "mike", null, false);
|
||||
nodesMetada[2] = getNodeMetaData(nodes[2], txn, acl, "mike", null, false);
|
||||
nodesMetada[3] = getNodeMetaData(nodes[3], txn, acl, "mike", null, false);
|
||||
NODES_METADATA[0] = getNodeMetaData(NODES[0], txn, ACL, "mike", null, false);
|
||||
NODES_METADATA[1] = getNodeMetaData(NODES[1], txn, ACL, "mike", null, false);
|
||||
NODES_METADATA[2] = getNodeMetaData(NODES[2], txn, ACL, "mike", null, false);
|
||||
NODES_METADATA[3] = getNodeMetaData(NODES[3], txn, ACL, "mike", null, false);
|
||||
|
||||
List<String> content = new ArrayList();
|
||||
List<String> content = new ArrayList<>();
|
||||
int[] sizes = {2000, 1000, 1500, 750};
|
||||
|
||||
Random r = new Random(1);
|
||||
@@ -127,16 +120,16 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
if(s>0) {
|
||||
buf.append(" ");
|
||||
}
|
||||
buf.append(Integer.toString(Math.abs(rand.nextInt())));
|
||||
buf.append(Math.abs(rand.nextInt()));
|
||||
}
|
||||
content.add(buf.toString());
|
||||
}
|
||||
|
||||
//Index the transaction, nodes, and nodeMetaDatas.
|
||||
//Index the transaction, NODES, and nodeMetaDatas.
|
||||
//Note that the content is automatically created by the test framework.
|
||||
indexTransaction(txn,
|
||||
list(nodes[0], nodes[1], nodes[2], nodes[3]),
|
||||
list(nodesMetada[0], nodesMetada[1], nodesMetada[2], nodesMetada[3]),
|
||||
asList(NODES[0], NODES[1], NODES[2], NODES[3]),
|
||||
asList(NODES_METADATA[0], NODES_METADATA[1], NODES_METADATA[2], NODES_METADATA[3]),
|
||||
content);
|
||||
|
||||
//Check for the TXN state stamp.
|
||||
@@ -156,7 +149,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
}
|
||||
|
||||
@AfterClass
|
||||
private static void destroyData() throws Throwable
|
||||
private static void destroyData()
|
||||
{
|
||||
dismissSolrServers();
|
||||
}
|
||||
@@ -167,7 +160,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
putHandleDefaults();
|
||||
QueryResponse response = query(getDefaultTestClient(), true,
|
||||
"{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}",
|
||||
params("q", "FINGERPRINT:"+nodes[0].getId(),
|
||||
params("q", "FINGERPRINT:"+ NODES[0].getId(),
|
||||
"qt", "/afts",
|
||||
"shards.qt", "/afts",
|
||||
"start", "0",
|
||||
@@ -175,22 +168,22 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
"rows", "100"));
|
||||
|
||||
SolrDocumentList docs = response.getResults();
|
||||
assertTrue(docs.getNumFound() == 4);
|
||||
assertEquals(4, docs.getNumFound());
|
||||
SolrDocument doc0 = docs.get(0);
|
||||
long dbid0 = (long)doc0.getFieldValue("DBID");
|
||||
assertTrue(dbid0 == nodes[0].getId());
|
||||
assertEquals(dbid0, NODES[0].getId());
|
||||
|
||||
SolrDocument doc1 = docs.get(1);
|
||||
long dbid1 = (long)doc1.getFieldValue("DBID");
|
||||
assertTrue(dbid1 == nodes[2].getId());
|
||||
assertEquals(dbid1, NODES[2].getId());
|
||||
|
||||
SolrDocument doc2 = docs.get(2);
|
||||
long dbid2 = (long)doc2.getFieldValue("DBID");
|
||||
assertTrue(dbid2 == nodes[1].getId());
|
||||
assertEquals(dbid2, NODES[1].getId());
|
||||
|
||||
SolrDocument doc3 = docs.get(3);
|
||||
long dbid3 = (long)doc3.getFieldValue("DBID");
|
||||
assertTrue(dbid3 == nodes[3].getId());
|
||||
assertEquals(dbid3, NODES[3].getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -199,7 +192,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
putHandleDefaults();
|
||||
QueryResponse response = query(getDefaultTestClient(), true,
|
||||
"{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}",
|
||||
params("q", "FINGERPRINT:" + nodes[0].getId()+"_70",
|
||||
params("q", "FINGERPRINT:" + NODES[0].getId()+"_70",
|
||||
"qt", "/afts",
|
||||
"shards.qt", "/afts",
|
||||
"start", "0",
|
||||
@@ -207,14 +200,14 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
"rows", "100"));
|
||||
|
||||
SolrDocumentList docs = response.getResults();
|
||||
assertTrue(docs.getNumFound() == 2);
|
||||
assertEquals(2, docs.getNumFound());
|
||||
SolrDocument doc0 = docs.get(0);
|
||||
long dbid0 = (long)doc0.getFieldValue("DBID");
|
||||
assertTrue(dbid0 == nodes[0].getId());
|
||||
assertEquals(dbid0, NODES[0].getId());
|
||||
|
||||
SolrDocument doc1 = docs.get(1);
|
||||
long dbid1 = (long)doc1.getFieldValue("DBID");
|
||||
assertTrue(dbid1 == nodes[2].getId());
|
||||
assertEquals(dbid1, NODES[2].getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -223,7 +216,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
putHandleDefaults();
|
||||
QueryResponse response = query(getDefaultTestClient(), true,
|
||||
"{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}",
|
||||
params("q", "FINGERPRINT:" + nodes[0].getId()+"_45",
|
||||
params("q", "FINGERPRINT:" + NODES[0].getId()+"_45",
|
||||
"qt", "/afts",
|
||||
"shards.qt", "/afts",
|
||||
"start", "0",
|
||||
@@ -231,18 +224,18 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
"rows", "100"));
|
||||
|
||||
SolrDocumentList docs = response.getResults();
|
||||
assertTrue(docs.getNumFound() == 3);
|
||||
assertEquals(3, docs.getNumFound());
|
||||
SolrDocument doc0 = docs.get(0);
|
||||
long dbid0 = (long)doc0.getFieldValue("DBID");
|
||||
assertTrue(dbid0 == nodes[0].getId());
|
||||
assertEquals(dbid0, NODES[0].getId());
|
||||
|
||||
SolrDocument doc1 = docs.get(1);
|
||||
long dbid1 = (long)doc1.getFieldValue("DBID");
|
||||
assertTrue(dbid1 == nodes[2].getId());
|
||||
assertEquals(dbid1, NODES[2].getId());
|
||||
|
||||
SolrDocument doc2 = docs.get(2);
|
||||
long dbid2 = (long)doc2.getFieldValue("DBID");
|
||||
assertTrue(dbid2 == nodes[1].getId());
|
||||
assertEquals(dbid2, NODES[1].getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -251,7 +244,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
putHandleDefaults();
|
||||
QueryResponse response = query(getDefaultTestClient(), true,
|
||||
"{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}",
|
||||
params("q", "FINGERPRINT:"+nodesMetada[0].getNodeRef().getId(),
|
||||
params("q", "FINGERPRINT:"+ NODES_METADATA[0].getNodeRef().getId(),
|
||||
"qt", "/afts",
|
||||
"shards.qt", "/afts",
|
||||
"start", "0",
|
||||
@@ -259,22 +252,22 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
"rows", "100"));
|
||||
|
||||
SolrDocumentList docs = response.getResults();
|
||||
assertTrue(docs.getNumFound() == 4);
|
||||
assertEquals(4, docs.getNumFound());
|
||||
SolrDocument doc0 = docs.get(0);
|
||||
long dbid0 = (long)doc0.getFieldValue("DBID");
|
||||
assertTrue(dbid0 == nodes[0].getId());
|
||||
assertEquals(dbid0, NODES[0].getId());
|
||||
|
||||
SolrDocument doc1 = docs.get(1);
|
||||
long dbid1 = (long)doc1.getFieldValue("DBID");
|
||||
assertTrue(dbid1 == nodes[2].getId());
|
||||
assertEquals(dbid1, NODES[2].getId());
|
||||
|
||||
SolrDocument doc2 = docs.get(2);
|
||||
long dbid2 = (long)doc2.getFieldValue("DBID");
|
||||
assertTrue(dbid2 == nodes[1].getId());
|
||||
assertEquals(dbid2, NODES[1].getId());
|
||||
|
||||
SolrDocument doc3 = docs.get(3);
|
||||
long dbid3 = (long)doc3.getFieldValue("DBID");
|
||||
assertTrue(dbid3 == nodes[3].getId());
|
||||
assertEquals(dbid3, NODES[3].getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -283,7 +276,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
putHandleDefaults();
|
||||
QueryResponse response = query(getDefaultTestClient(), true,
|
||||
"{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}",
|
||||
params("q", "FINGERPRINT:" + nodesMetada[0].getNodeRef().getId() +"_70",
|
||||
params("q", "FINGERPRINT:" + NODES_METADATA[0].getNodeRef().getId() +"_70",
|
||||
"qt", "/afts",
|
||||
"shards.qt", "/afts",
|
||||
"start", "0",
|
||||
@@ -291,14 +284,14 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
"rows", "100"));
|
||||
|
||||
SolrDocumentList docs = response.getResults();
|
||||
assertTrue(docs.getNumFound() == 2);
|
||||
assertEquals(2, docs.getNumFound());
|
||||
SolrDocument doc0 = docs.get(0);
|
||||
long dbid0 = (long)doc0.getFieldValue("DBID");
|
||||
assertTrue(dbid0 == nodes[0].getId());
|
||||
assertEquals(dbid0, NODES[0].getId());
|
||||
|
||||
SolrDocument doc1 = docs.get(1);
|
||||
long dbid1 = (long)doc1.getFieldValue("DBID");
|
||||
assertTrue(dbid1 == nodes[2].getId());
|
||||
assertEquals(dbid1, NODES[2].getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -307,7 +300,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
putHandleDefaults();
|
||||
QueryResponse response = query(getDefaultTestClient(), true,
|
||||
"{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}",
|
||||
params("q", "FINGERPRINT:" + nodesMetada[0].getNodeRef().getId() +"_45",
|
||||
params("q", "FINGERPRINT:" + NODES_METADATA[0].getNodeRef().getId() +"_45",
|
||||
"qt", "/afts",
|
||||
"shards.qt", "/afts",
|
||||
"start", "0",
|
||||
@@ -315,20 +308,113 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist
|
||||
"rows", "100"));
|
||||
|
||||
SolrDocumentList docs = response.getResults();
|
||||
assertTrue(docs.getNumFound() == 3);
|
||||
assertEquals(3, docs.getNumFound());
|
||||
SolrDocument doc0 = docs.get(0);
|
||||
long dbid0 = (long)doc0.getFieldValue("DBID");
|
||||
assertTrue(dbid0 == nodes[0].getId());
|
||||
assertEquals(dbid0, NODES[0].getId());
|
||||
|
||||
SolrDocument doc1 = docs.get(1);
|
||||
long dbid1 = (long)doc1.getFieldValue("DBID");
|
||||
assertTrue(dbid1 == nodes[2].getId());
|
||||
assertEquals(dbid1, NODES[2].getId());
|
||||
|
||||
SolrDocument doc2 = docs.get(2);
|
||||
long dbid2 = (long)doc2.getFieldValue("DBID");
|
||||
assertTrue(dbid2 == nodes[1].getId());
|
||||
assertEquals(dbid2, NODES[1].getId());
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFingerprintStillExistsAfterNodeMetadataUpdate() throws Exception
|
||||
{
|
||||
putHandleDefaults();
|
||||
|
||||
Transaction txn = getTransaction(0, 1);
|
||||
Node fileNode = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED);
|
||||
NodeMetaData fileMetaData = getNodeMetaData(fileNode, txn, ACL, "mike", null, false);
|
||||
|
||||
indexTransaction(
|
||||
txn,
|
||||
singletonList(fileNode),
|
||||
singletonList(fileMetaData),
|
||||
singletonList("This is a text content which is longer than the default hello world " + fileNode.getId() +
|
||||
" returned by the Mock SOLRAPIQueueClient. This is needed because the \"min_hash\" field type " +
|
||||
"definition in Solr doesn't take in account fields which produce less than 5 tokens (see the " +
|
||||
"ShingleFilter settings)."));
|
||||
|
||||
makeSureContentNodeHasBeenIndexed(fileNode, "mike", "longer");
|
||||
|
||||
QueryResponse response = query(getDefaultTestClient(), true,
|
||||
"{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}",
|
||||
params("q", "FINGERPRINT:" + fileMetaData.getNodeRef().getId(),
|
||||
"qt", "/afts",
|
||||
"shards.qt", "/afts",
|
||||
"start", "0",
|
||||
"fl", "DBID,score",
|
||||
"rows", "100"));
|
||||
|
||||
SolrDocumentList docs = response.getResults();
|
||||
assertEquals(1, docs.getNumFound());
|
||||
assertEquals(fileNode.getId(), docs.iterator().next().getFieldValue("DBID"));
|
||||
|
||||
// Let's update the test node
|
||||
fileMetaData.setOwner("Andrea");
|
||||
fileMetaData.getProperties().put(ContentModel.PROP_TITLE, new StringPropertyValue("This is the new file \"title\" metadata attribute."));
|
||||
|
||||
txn = getTransaction(0, 1);
|
||||
|
||||
indexTransaction(
|
||||
txn,
|
||||
singletonList(fileNode),
|
||||
singletonList(fileMetaData));
|
||||
|
||||
makeSureContentNodeHasBeenIndexed(fileNode, "Andrea", "longer");
|
||||
|
||||
response = query(getDefaultTestClient(), true,
|
||||
"{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}",
|
||||
params("q", "FINGERPRINT:" + fileMetaData.getNodeRef().getId(),
|
||||
"qt", "/afts",
|
||||
"shards.qt", "/afts",
|
||||
"start", "0",
|
||||
"fl", "DBID,score",
|
||||
"rows", "100"));
|
||||
docs = response.getResults();
|
||||
assertEquals(1, docs.getNumFound());
|
||||
assertEquals(fileNode.getId(), docs.iterator().next().getFieldValue("DBID"));
|
||||
}
|
||||
|
||||
/**
|
||||
* Queries the index using a token from the (dummy) text produced by the test framework ("world", actually).
|
||||
* Once the query returns a positive result we are sure the ContentTracker
|
||||
*
|
||||
* <ol>
|
||||
* <li>
|
||||
* Fetched the text content associated with the current node, from Alfresco
|
||||
* </li>
|
||||
* <li>
|
||||
* Computed a fingerprint (using the retrieved text) for the node
|
||||
* </li>
|
||||
* <li>
|
||||
* Updated the node definition in the (Solr)ContentStore and in Solr
|
||||
* </li>
|
||||
* </ol>
|
||||
*
|
||||
* Last but not least, we are also making sure that CommitTracker executed its cycle as well (otherwise document
|
||||
* wouldn't be searchable).
|
||||
*
|
||||
* @param node an addition term which will be appended as a required clause in the executed query.
|
||||
* @param testTerm a term which is supposed to be in the indexed content
|
||||
* @param owner the #FIELD_OWNER which will be used as an additional required query clause.
|
||||
* @throws Exception in case the MAX_WAIT_TIME is reached and the node is not in results.
|
||||
*/
|
||||
private void makeSureContentNodeHasBeenIndexed(final Node node, final String owner, String testTerm) throws Exception
|
||||
{
|
||||
waitForDocCount(new TermQuery(new Term(QueryConstants.FIELD_READER, "jim")), 1, MAX_WAIT_TIME);
|
||||
waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), 1, MAX_WAIT_TIME);
|
||||
|
||||
waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", Long.toString(node.getId()))), 1, MAX_WAIT_TIME);
|
||||
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_OWNER, owner)), BooleanClause.Occur.MUST));
|
||||
waitForDocCount(builder.build(), 1, MAX_WAIT_TIME);
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -19,7 +19,7 @@
|
||||
package org.alfresco.solr.query.afts.qparser;
|
||||
|
||||
import org.alfresco.repo.search.adaptor.lucene.QueryConstants;
|
||||
import org.alfresco.solr.query.afts.TestDataProvider;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.alfresco.util.ISO9075;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
+1
-1
@@ -30,7 +30,7 @@ import static org.alfresco.model.ContentModel.TYPE_THUMBNAIL;
|
||||
|
||||
import org.alfresco.repo.search.adaptor.lucene.QueryConstants;
|
||||
import org.alfresco.service.namespace.QName;
|
||||
import org.alfresco.solr.query.afts.TestDataProvider;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.alfresco.util.CachingDateFormat;
|
||||
import org.apache.solr.SolrTestCaseJ4;
|
||||
import org.junit.BeforeClass;
|
||||
|
||||
+1
-1
@@ -22,7 +22,7 @@ import org.alfresco.service.cmr.repository.NodeRef;
|
||||
import org.alfresco.service.namespace.QName;
|
||||
import org.alfresco.solr.client.PropertyValue;
|
||||
import org.alfresco.solr.client.StringPropertyValue;
|
||||
import org.alfresco.solr.query.afts.TestDataProvider;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
|
||||
+1
-1
@@ -30,7 +30,7 @@ import org.alfresco.service.cmr.repository.NodeRef;
|
||||
import org.alfresco.service.namespace.QName;
|
||||
import org.alfresco.solr.client.PropertyValue;
|
||||
import org.alfresco.solr.client.StringPropertyValue;
|
||||
import org.alfresco.solr.query.afts.TestDataProvider;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
|
||||
+1
-1
@@ -27,7 +27,7 @@ import org.alfresco.service.cmr.repository.NodeRef;
|
||||
import org.alfresco.service.namespace.QName;
|
||||
import org.alfresco.solr.client.PropertyValue;
|
||||
import org.alfresco.solr.client.StringPropertyValue;
|
||||
import org.alfresco.solr.query.afts.TestDataProvider;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ import org.alfresco.service.cmr.repository.NodeRef;
|
||||
import org.alfresco.service.namespace.QName;
|
||||
import org.alfresco.solr.client.PropertyValue;
|
||||
import org.alfresco.solr.client.StringPropertyValue;
|
||||
import org.alfresco.solr.query.afts.TestDataProvider;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.alfresco.util.Pair;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
+1
-1
@@ -10,7 +10,7 @@ import org.alfresco.repo.search.adaptor.lucene.QueryConstants;
|
||||
import org.alfresco.service.namespace.QName;
|
||||
import org.alfresco.solr.AlfrescoSolrDataModel;
|
||||
import org.alfresco.solr.SolrInformationServer;
|
||||
import org.alfresco.solr.query.afts.TestDataProvider;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.apache.lucene.util.LuceneTestCase;
|
||||
import org.apache.solr.SolrTestCaseJ4;
|
||||
import org.junit.BeforeClass;
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
package org.alfresco.solr.query.afts.requestHandler;
|
||||
|
||||
import org.alfresco.solr.query.afts.TestDataProvider;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
|
||||
+197
@@ -0,0 +1,197 @@
|
||||
/*
|
||||
* Copyright (C) 2005-2019 Alfresco Software Limited.
|
||||
*
|
||||
* This file is part of Alfresco
|
||||
*
|
||||
* Alfresco is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU Lesser General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* Alfresco is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General Public License
|
||||
* along with Alfresco. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
package org.alfresco.solr.tracker;
|
||||
|
||||
import static java.util.Arrays.asList;
|
||||
import static java.util.Collections.singletonList;
|
||||
import static java.util.Optional.of;
|
||||
import static java.util.stream.IntStream.range;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.MAX_WAIT_TIME;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.assertShardAndCoreSummaryConsistency;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.coreAdminHandler;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAcl;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAclChangeSet;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAclReaders;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getTransaction;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.indexAclChangeSet;
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertNotEquals;
|
||||
|
||||
import org.alfresco.repo.index.shard.ShardState;
|
||||
import org.alfresco.repo.search.adaptor.lucene.QueryConstants;
|
||||
import org.alfresco.solr.AbstractAlfrescoSolrTests;
|
||||
import org.alfresco.solr.AlfrescoCoreAdminHandler;
|
||||
import org.alfresco.solr.client.Acl;
|
||||
import org.alfresco.solr.client.AclChangeSet;
|
||||
import org.alfresco.solr.client.AclReaders;
|
||||
import org.alfresco.solr.client.Node;
|
||||
import org.alfresco.solr.client.NodeMetaData;
|
||||
import org.alfresco.solr.client.SOLRAPIQueueClient;
|
||||
import org.alfresco.solr.client.Transaction;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.apache.lucene.index.Term;
|
||||
import org.apache.lucene.search.BooleanClause;
|
||||
import org.apache.lucene.search.BooleanQuery;
|
||||
import org.apache.lucene.search.LegacyNumericRangeQuery;
|
||||
import org.apache.lucene.search.TermQuery;
|
||||
import org.apache.solr.core.SolrCore;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
/**
|
||||
* A partial state of {@link org.alfresco.solr.TrackerState} is exposed through two interfaces: AdminHandler.SUMMARY and
|
||||
* {@link MetadataTracker#getShardState}.
|
||||
*
|
||||
* This test makes sure that state is consistent across the two mentioned approaches. That is, properties returned by the
|
||||
* Core SUMMARY must have the same value of the same properties in the ShardState.
|
||||
*
|
||||
* @author agazzarini
|
||||
*/
|
||||
public class AlfrescoSolrTrackerStateTest extends AbstractAlfrescoSolrTests
|
||||
{
|
||||
@BeforeClass
|
||||
public static void beforeClass() throws Exception
|
||||
{
|
||||
initAlfrescoCore("schema.xml");
|
||||
}
|
||||
|
||||
@After
|
||||
public void clearQueue() throws Exception {
|
||||
SOLRAPIQueueClient.nodeMetaDataMap.clear();
|
||||
SOLRAPIQueueClient.transactionQueue.clear();
|
||||
SOLRAPIQueueClient.aclChangeSetQueue.clear();
|
||||
SOLRAPIQueueClient.aclReadersMap.clear();
|
||||
SOLRAPIQueueClient.aclMap.clear();
|
||||
SOLRAPIQueueClient.nodeMap.clear();
|
||||
}
|
||||
|
||||
@Before
|
||||
public void indexTestAclTransactionAndNodes() throws Exception
|
||||
{
|
||||
Acl acl = createAndIndexSomeAclData();
|
||||
createAndIndexTransactionWithSomeNodes(4, acl, "first");
|
||||
}
|
||||
|
||||
@Test
|
||||
@SuppressWarnings("deprecated")
|
||||
public void shardStateMustBeConsistentWithCoreSummaryStats() throws Exception {
|
||||
SolrCore core = getCore();
|
||||
|
||||
MetadataTracker tracker =
|
||||
of(coreAdminHandler(core))
|
||||
.map(AlfrescoCoreAdminHandler::getTrackerRegistry)
|
||||
.map(registry -> registry.getTrackerForCore(core.getName(), MetadataTracker.class))
|
||||
.orElseThrow(() -> new IllegalStateException("Cannot retrieve the Metadata tracker on this test core."));
|
||||
|
||||
// 1. First consistency check: ShardState must have the same values of CoreAdmin.SUMMARY report
|
||||
ShardState shardStateAfterFirstIndexingRound = tracker.getShardState();
|
||||
assertShardAndCoreSummaryConsistency(shardStateAfterFirstIndexingRound, core);
|
||||
|
||||
// 2. Index additional ACLs
|
||||
Acl acl = createAndIndexSomeAclData();
|
||||
|
||||
ShardState shardStateAfterIndexingSomeAdditionalAcl= tracker.getShardState();
|
||||
|
||||
// 3. We indexed only ACLs, so ACL data must be different while Transaction data must be the same
|
||||
assertEquals(shardStateAfterFirstIndexingRound.getLastIndexedTxId(), shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedTxId());
|
||||
assertEquals(shardStateAfterFirstIndexingRound.getLastIndexedTxCommitTime(), shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedTxCommitTime());
|
||||
assertNotEquals(shardStateAfterFirstIndexingRound.getLastIndexedChangeSetId(), shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedChangeSetId());
|
||||
assertNotEquals(shardStateAfterFirstIndexingRound.getLastIndexedChangeSetCommitTime(), shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedChangeSetCommitTime());
|
||||
|
||||
// Second consistency check: ShardState must have the same values of CoreAdmin.SUMMARY report
|
||||
assertShardAndCoreSummaryConsistency(shardStateAfterIndexingSomeAdditionalAcl, core);
|
||||
|
||||
// 4. Index a transaction with 5 other nodes
|
||||
createAndIndexTransactionWithSomeNodes(5, acl, "second");
|
||||
|
||||
ShardState shardStateAfterIndexingAnAdditionalTransaction = tracker.getShardState();
|
||||
assertNotEquals(shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedTxId(), shardStateAfterIndexingAnAdditionalTransaction.getLastIndexedTxId());
|
||||
assertNotEquals(shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedTxCommitTime(), shardStateAfterIndexingAnAdditionalTransaction.getLastIndexedTxCommitTime());
|
||||
assertEquals(shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedChangeSetId(), shardStateAfterIndexingAnAdditionalTransaction.getLastIndexedChangeSetId());
|
||||
assertEquals(shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedChangeSetCommitTime(), shardStateAfterIndexingAnAdditionalTransaction.getLastIndexedChangeSetCommitTime());
|
||||
|
||||
// 5. Third consistency check: ShardState must have the same values of CoreAdmin.SUMMARY report
|
||||
assertShardAndCoreSummaryConsistency(tracker.getShardState(), core);
|
||||
}
|
||||
|
||||
private void makeSureTransactionHasBeenIndexed(long transactionId) throws Exception
|
||||
{
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!TX")), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_TXID, transactionId, transactionId + 1, true, false), BooleanClause.Occur.MUST));
|
||||
BooleanQuery waitForQuery = builder.build();
|
||||
waitForDocCount(waitForQuery, 1, MAX_WAIT_TIME);
|
||||
}
|
||||
|
||||
private void makeSureNodesHaveBeenIndexed(int expectedCount, String searchText) throws Exception
|
||||
{
|
||||
waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", searchText)), expectedCount, MAX_WAIT_TIME);
|
||||
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", searchText)), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_OWNER, "mike")), BooleanClause.Occur.MUST));
|
||||
waitForDocCount(builder.build(), expectedCount, MAX_WAIT_TIME);
|
||||
}
|
||||
|
||||
private Acl createAndIndexSomeAclData() throws Exception
|
||||
{
|
||||
AclChangeSet aclChangeSet = getAclChangeSet(1);
|
||||
|
||||
Acl acl = getAcl(aclChangeSet);
|
||||
Acl acl2 = getAcl(aclChangeSet);
|
||||
|
||||
AclReaders aclReaders = getAclReaders(aclChangeSet, acl, singletonList("joel"), singletonList("phil"), null);
|
||||
AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, singletonList("jim"), singletonList("phil"), null);
|
||||
|
||||
indexAclChangeSet(aclChangeSet, asList(acl, acl2), asList(aclReaders, aclReaders2));
|
||||
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!ACLTX")), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_ACLTXID, aclChangeSet.getId(), aclChangeSet.getId() + 1, true, false), BooleanClause.Occur.MUST));
|
||||
BooleanQuery waitForQuery = builder.build();
|
||||
waitForDocCount(waitForQuery, 1, MAX_WAIT_TIME);
|
||||
|
||||
return acl;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates and indexes a transaction with a certain number of nodes.
|
||||
*
|
||||
* @param howManyTestNodes how many nodes we want to index.
|
||||
* @param acl the related ACL.
|
||||
* @param sampleTextContent a sample text content that will be used to assert nodes have been actually indexed.
|
||||
*/
|
||||
private void createAndIndexTransactionWithSomeNodes(int howManyTestNodes, Acl acl, String sampleTextContent) throws Exception
|
||||
{
|
||||
Transaction txn = getTransaction(0, howManyTestNodes);
|
||||
Map.Entry<List<Node>, List<NodeMetaData>> data = TestDataProvider.nSampleNodesWithSampleContent(acl, txn, howManyTestNodes);
|
||||
|
||||
indexTransaction(txn, data.getKey(), data.getValue(), range(0, howManyTestNodes).mapToObj(index -> sampleTextContent).collect(Collectors.toList()));
|
||||
|
||||
makeSureTransactionHasBeenIndexed(txn.getId());
|
||||
makeSureNodesHaveBeenIndexed(data.getKey().size(), sampleTextContent);
|
||||
}
|
||||
}
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
/*
|
||||
* Copyright (C) 2005-2019 Alfresco Software Limited.
|
||||
*
|
||||
* This file is part of Alfresco
|
||||
*
|
||||
* Alfresco is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU Lesser General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* Alfresco is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General Public License
|
||||
* along with Alfresco. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
package org.alfresco.solr.tracker;
|
||||
|
||||
import static java.util.Arrays.asList;
|
||||
import static java.util.Collections.singletonList;
|
||||
import static java.util.Optional.of;
|
||||
import static java.util.stream.IntStream.range;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.MAX_WAIT_TIME;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.assertShardAndCoreSummaryConsistency;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.coreAdminHandler;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAcl;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAclChangeSet;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAclReaders;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getTransaction;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.indexAclChangeSet;
|
||||
|
||||
import org.alfresco.repo.index.shard.ShardState;
|
||||
import org.alfresco.repo.search.adaptor.lucene.QueryConstants;
|
||||
import org.alfresco.solr.AbstractAlfrescoDistributedTest;
|
||||
import org.alfresco.solr.AlfrescoCoreAdminHandler;
|
||||
import org.alfresco.solr.client.Acl;
|
||||
import org.alfresco.solr.client.AclChangeSet;
|
||||
import org.alfresco.solr.client.AclReaders;
|
||||
import org.alfresco.solr.client.Node;
|
||||
import org.alfresco.solr.client.NodeMetaData;
|
||||
import org.alfresco.solr.client.Transaction;
|
||||
import org.alfresco.solr.dataload.TestDataProvider;
|
||||
import org.apache.lucene.document.LongPoint;
|
||||
import org.apache.lucene.index.Term;
|
||||
import org.apache.lucene.search.BooleanClause;
|
||||
import org.apache.lucene.search.BooleanQuery;
|
||||
import org.apache.lucene.search.LegacyNumericRangeQuery;
|
||||
import org.apache.lucene.search.TermQuery;
|
||||
import org.apache.solr.SolrTestCaseJ4;
|
||||
import org.junit.AfterClass;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
/**
|
||||
* A partial state of {@link org.alfresco.solr.TrackerState} is exposed through two interfaces: AdminHandler.SUMMARY and
|
||||
* {@link MetadataTracker#getShardState}.
|
||||
* This test makes sure that state is consistent across the two mentioned approaches. That is, properties returned by the
|
||||
* Core SUMMARY must have the same value of the same properties in the ShardState.
|
||||
*
|
||||
* Note that this is the distributed version of {@link AlfrescoSolrTrackerStateTest}.
|
||||
*
|
||||
* @author agazzarini
|
||||
*/
|
||||
@SolrTestCaseJ4.SuppressSSL
|
||||
public class DistributedAlfrescoSolrTrackerStateTest extends AbstractAlfrescoDistributedTest
|
||||
{
|
||||
@BeforeClass
|
||||
private static void initData() throws Throwable
|
||||
{
|
||||
initSolrServers(5, getClassName(),null);
|
||||
|
||||
Acl acl = createAndIndexSomeAclData();
|
||||
createAndIndexTransactionWithSomeNodes(5, acl, "first");
|
||||
}
|
||||
|
||||
@AfterClass
|
||||
private static void destroyData()
|
||||
{
|
||||
dismissSolrServers();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void shardStateMustBeConsistentWithCoreSummaryStats()
|
||||
{
|
||||
putHandleDefaults();
|
||||
|
||||
getJettyCores(solrShards).forEach(core -> {
|
||||
MetadataTracker tracker =
|
||||
of(coreAdminHandler(core))
|
||||
.map(AlfrescoCoreAdminHandler::getTrackerRegistry)
|
||||
.map(registry -> registry.getTrackerForCore(core.getName(), MetadataTracker.class))
|
||||
.orElseThrow(() -> new IllegalStateException("Cannot retrieve the Metadata tracker on this test core."));
|
||||
|
||||
// 1. First consistency check: ShardState must have the same values of CoreAdmin.SUMMARY report
|
||||
ShardState shardStateAfterFirstIndexingRound = tracker.getShardState();
|
||||
assertShardAndCoreSummaryConsistency(shardStateAfterFirstIndexingRound, core);
|
||||
|
||||
// 2. Index additional ACLs
|
||||
Acl acl = createAndIndexSomeAclData();
|
||||
|
||||
ShardState shardStateAfterIndexingSomeAdditionalAcl= tracker.getShardState();
|
||||
|
||||
// 3. We indexed only ACLs, so ACL data must be different while Transaction data must be the same
|
||||
assertEquals(shardStateAfterFirstIndexingRound.getLastIndexedTxId(), shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedTxId());
|
||||
assertEquals(shardStateAfterFirstIndexingRound.getLastIndexedTxCommitTime(), shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedTxCommitTime());
|
||||
assertNotEquals(shardStateAfterFirstIndexingRound.getLastIndexedChangeSetId(), shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedChangeSetId());
|
||||
assertNotEquals(shardStateAfterFirstIndexingRound.getLastIndexedChangeSetCommitTime(), shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedChangeSetCommitTime());
|
||||
|
||||
// Second consistency check: ShardState must have the same values of CoreAdmin.SUMMARY report
|
||||
assertShardAndCoreSummaryConsistency(shardStateAfterIndexingSomeAdditionalAcl, core);
|
||||
|
||||
// 4. Index a transaction with 10 other nodes
|
||||
createAndIndexTransactionWithSomeNodes(10, acl, "second");
|
||||
|
||||
ShardState shardStateAfterIndexingAnAdditionalTransaction = tracker.getShardState();
|
||||
assertNotEquals(shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedTxId(), shardStateAfterIndexingAnAdditionalTransaction.getLastIndexedTxId());
|
||||
assertNotEquals(shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedTxCommitTime(), shardStateAfterIndexingAnAdditionalTransaction.getLastIndexedTxCommitTime());
|
||||
assertEquals(shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedChangeSetId(), shardStateAfterIndexingAnAdditionalTransaction.getLastIndexedChangeSetId());
|
||||
assertEquals(shardStateAfterIndexingSomeAdditionalAcl.getLastIndexedChangeSetCommitTime(), shardStateAfterIndexingAnAdditionalTransaction.getLastIndexedChangeSetCommitTime());
|
||||
|
||||
// 5. Third consistency check: ShardState must have the same values of CoreAdmin.SUMMARY report
|
||||
assertShardAndCoreSummaryConsistency(tracker.getShardState(), core);
|
||||
});
|
||||
}
|
||||
|
||||
private static Acl createAndIndexSomeAclData()
|
||||
{
|
||||
try {
|
||||
|
||||
AclChangeSet aclChangeSet = getAclChangeSet(1);
|
||||
|
||||
Acl acl = getAcl(aclChangeSet);
|
||||
Acl acl2 = getAcl(aclChangeSet);
|
||||
|
||||
AclReaders aclReaders = getAclReaders(aclChangeSet, acl, singletonList("joel"), singletonList("phil"), null);
|
||||
AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, singletonList("jim"), singletonList("phil"), null);
|
||||
|
||||
indexAclChangeSet(aclChangeSet, asList(acl, acl2), asList(aclReaders, aclReaders2));
|
||||
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!ACLTX")), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_ACLTXID, aclChangeSet.getId(), aclChangeSet.getId() + 1, true, false), BooleanClause.Occur.MUST));
|
||||
BooleanQuery waitForQuery = builder.build();
|
||||
waitForDocCountAllCores(waitForQuery, 1, MAX_WAIT_TIME);
|
||||
|
||||
return acl;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
throw new RuntimeException(exception);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates and indexes a transaction with a certain number of nodes.
|
||||
*
|
||||
* @param howManyTestNodes how many nodes we want to index.
|
||||
* @param acl the related ACL.
|
||||
* @param sampleTextContent a sample text content that will be used to assert nodes have been actually indexed.
|
||||
*/
|
||||
private static void createAndIndexTransactionWithSomeNodes(int howManyTestNodes, Acl acl, String sampleTextContent)
|
||||
{
|
||||
try
|
||||
{
|
||||
Transaction txn = getTransaction(0, howManyTestNodes);
|
||||
Map.Entry<List<Node>, List<NodeMetaData>> data = TestDataProvider.nSampleNodesWithSampleContent(acl, txn, howManyTestNodes);
|
||||
|
||||
indexTransaction(txn, data.getKey(), data.getValue(), range(0, howManyTestNodes).mapToObj(index -> sampleTextContent).collect(Collectors.toList()));
|
||||
|
||||
BooleanQuery.Builder builder = new BooleanQuery.Builder();
|
||||
builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!TX")), BooleanClause.Occur.MUST));
|
||||
builder.add(new BooleanClause(LongPoint.newExactQuery(QueryConstants.FIELD_S_TXID, txn.getId()), BooleanClause.Occur.MUST));
|
||||
BooleanQuery waitForQuery = builder.build();
|
||||
|
||||
waitForDocCountAllCores(waitForQuery, 1, MAX_WAIT_TIME);
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
throw new RuntimeException(exception);
|
||||
}
|
||||
}
|
||||
}
|
||||
+203
@@ -0,0 +1,203 @@
|
||||
/*
|
||||
* Copyright (C) 2005-2019 Alfresco Software Limited.
|
||||
*
|
||||
* This file is part of Alfresco
|
||||
*
|
||||
* Alfresco is free software: you can redistribute it and/or modify
|
||||
* it under the terms of the GNU Lesser General Public License as published by
|
||||
* the Free Software Foundation, either version 3 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* Alfresco is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU Lesser General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU Lesser General Public License
|
||||
* along with Alfresco. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
package org.alfresco.solr.tracker;
|
||||
|
||||
import static java.util.Collections.singletonList;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.ancestors;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAcl;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAclChangeSet;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getAclReaders;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getNode;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getNodeMetaData;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.getTransaction;
|
||||
import static org.alfresco.solr.AlfrescoSolrUtils.indexAclChangeSet;
|
||||
import static org.carrot2.shaded.guava.common.collect.ImmutableList.of;
|
||||
|
||||
import org.alfresco.model.ContentModel;
|
||||
import org.alfresco.solr.AbstractAlfrescoDistributedTest;
|
||||
import org.alfresco.solr.client.Acl;
|
||||
import org.alfresco.solr.client.AclChangeSet;
|
||||
import org.alfresco.solr.client.AclReaders;
|
||||
import org.alfresco.solr.client.Node;
|
||||
import org.alfresco.solr.client.NodeMetaData;
|
||||
import org.alfresco.solr.client.StringPropertyValue;
|
||||
import org.alfresco.solr.client.Transaction;
|
||||
import org.alfresco.util.Pair;
|
||||
import org.apache.lucene.index.Term;
|
||||
import org.apache.lucene.search.TermQuery;
|
||||
|
||||
import org.apache.lucene.util.LuceneTestCase;
|
||||
import org.apache.solr.SolrTestCaseJ4;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
import java.util.Properties;
|
||||
|
||||
/**
|
||||
* @author Elia
|
||||
* Test cascade tracker in multi sharded environment.
|
||||
*/
|
||||
@SolrTestCaseJ4.SuppressSSL
|
||||
@LuceneTestCase.SuppressCodecs({"Appending","Lucene3x","Lucene40","Lucene41","Lucene42","Lucene43", "Lucene44", "Lucene45","Lucene46","Lucene47","Lucene48","Lucene49"})
|
||||
public class DistributedCascadeTrackerTest extends AbstractAlfrescoDistributedTest
|
||||
{
|
||||
private Node parentFolder;
|
||||
private NodeMetaData parentFolderMetadata;
|
||||
|
||||
private Node childShard0;
|
||||
private NodeMetaData childShardMetadata0;
|
||||
|
||||
private Node childShard1;
|
||||
private NodeMetaData childShardMetadata1;
|
||||
|
||||
private final String pathParent = "pathParent";
|
||||
private final String pathChild0 = "pathChild0";
|
||||
private final String pathChild1 = "pathChild2";
|
||||
|
||||
|
||||
private final int timeout = 100000;
|
||||
|
||||
@Before
|
||||
private void initData() throws Throwable
|
||||
{
|
||||
initSolrServers(2, getClassName(), getShardMethod());
|
||||
indexData();
|
||||
}
|
||||
|
||||
@After
|
||||
private void destroyData()
|
||||
{
|
||||
dismissSolrServers();
|
||||
}
|
||||
|
||||
/**
|
||||
* Default data is indexed in solr.
|
||||
* 1 folder node with 2 children nodes.
|
||||
* 1 Child is on the same shard of the parent folder (shard 0) while the other is on shard 1.
|
||||
*/
|
||||
private void indexData() throws Exception
|
||||
{
|
||||
AclChangeSet aclChangeSet = getAclChangeSet(1);
|
||||
|
||||
Acl acl = getAcl(aclChangeSet);
|
||||
AclReaders aclReaders = getAclReaders(aclChangeSet, acl, singletonList("joel"), singletonList("phil"), null);
|
||||
|
||||
indexAclChangeSet(aclChangeSet,
|
||||
of(acl),
|
||||
of(aclReaders));
|
||||
|
||||
indexNodes(acl);
|
||||
}
|
||||
|
||||
/**
|
||||
* This test checks if after updating the parent folder,
|
||||
* both the children(in both the shards) are updated as well in cascading.
|
||||
*/
|
||||
@Test
|
||||
public void testCascadeShouldHappenInBothShardsAfterUpdateParentFolder() throws Exception
|
||||
{
|
||||
|
||||
String cascadingFirstChild = "cascadingFirstChild";
|
||||
String cascadingSecondChild = "cascadingSecondChild";
|
||||
|
||||
/*
|
||||
* Modify children paths in order to see if they are updated both after cascading.
|
||||
*/
|
||||
childShardMetadata0.setPaths(of(new Pair<>(cascadingFirstChild, null)));
|
||||
childShardMetadata1.setPaths(of(new Pair<>(cascadingSecondChild, null)));
|
||||
|
||||
/*
|
||||
* Check the path of the two nodes before cascading
|
||||
*/
|
||||
assertShardCount(0, params("qt", "/afts", "q", "PATH:" + pathChild0), 1);
|
||||
assertShardCount(1, params("qt", "/afts", "q", "PATH:" + pathChild1), 1);
|
||||
|
||||
/*
|
||||
* Index a transaction with the parent folder update
|
||||
*/
|
||||
indexParentFolderWithCascade();
|
||||
|
||||
waitForDocCount(params("qt", "/afts", "q", "PATH:" + cascadingFirstChild), 1, timeout);
|
||||
|
||||
// Check if the path is updated for both the nodes
|
||||
assertShardCount(0, params("qt", "/afts", "q", "PATH:" + cascadingFirstChild), 1);
|
||||
assertShardCount(1, params("qt", "/afts", "q", "PATH:" + cascadingSecondChild), 1);
|
||||
|
||||
// The old paths are no longer in the index
|
||||
assertShardCount(0, params("qt", "/afts", "q", "PATH:" + pathChild0), 0);
|
||||
assertShardCount(1, params("qt", "/afts", "q", "PATH:" + pathChild1), 0);
|
||||
}
|
||||
|
||||
private void indexNodes(Acl acl) throws Exception
|
||||
{
|
||||
Transaction bigTxn = getTransaction(0, 3);
|
||||
|
||||
/*
|
||||
* Create parent folder in the first shard
|
||||
*/
|
||||
parentFolder = getNode(0, bigTxn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
parentFolderMetadata = getNodeMetaData(parentFolder, bigTxn, acl, "elia", null, false);
|
||||
parentFolderMetadata.setPaths(of(new Pair<>(pathParent, null)));
|
||||
|
||||
/*
|
||||
* Create first node.
|
||||
* This will be stored in the first shard (range [0-100])
|
||||
*/
|
||||
childShard0 = getNode(99, bigTxn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
childShardMetadata0 = getNodeMetaData(childShard0, bigTxn, acl, "elia", ancestors(parentFolderMetadata.getNodeRef()), false);
|
||||
childShardMetadata0.setPaths(of(new Pair<>(pathChild0, null)));
|
||||
|
||||
/*
|
||||
* Create second node.
|
||||
* This will be stored in the second shard (range [101-200])
|
||||
*/
|
||||
childShard1 = getNode(101, bigTxn, acl, Node.SolrApiNodeStatus.UPDATED);
|
||||
childShardMetadata1 = getNodeMetaData(childShard1, bigTxn, acl, "elia", ancestors(parentFolderMetadata.getNodeRef()), false);
|
||||
childShardMetadata1.setPaths(of(new Pair<>(pathChild1, null)));
|
||||
|
||||
indexTransaction(bigTxn,
|
||||
of(parentFolder, childShard0, childShard1),
|
||||
of(parentFolderMetadata, childShardMetadata0, childShardMetadata1));
|
||||
|
||||
/*
|
||||
* Get sure the nodes are indexed correctly in the shards
|
||||
*/
|
||||
waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", "world")), 3, timeout);
|
||||
assertShardCount(0, new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", "world")), 2);
|
||||
assertShardCount(1, new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", "world")), 1);
|
||||
}
|
||||
|
||||
private void indexParentFolderWithCascade()
|
||||
{
|
||||
Transaction bigTxn = getTransaction(0, 1);
|
||||
|
||||
parentFolder.setTxnId(bigTxn.getId());
|
||||
parentFolderMetadata.setTxnId(bigTxn.getId());
|
||||
parentFolderMetadata.getProperties().put(ContentModel.PROP_CASCADE_TX, new StringPropertyValue(Long.toString(bigTxn.getId())));
|
||||
|
||||
indexTransaction(bigTxn, of(parentFolder), of(parentFolderMetadata));
|
||||
}
|
||||
|
||||
private Properties getShardMethod()
|
||||
{
|
||||
Properties prop = new Properties();
|
||||
prop.put("shard.method", "DB_ID_RANGE");
|
||||
return prop;
|
||||
}
|
||||
}
|
||||
+2
@@ -118,7 +118,9 @@ public class DistributedDateMonthAlfrescoSolrTrackerTest extends AbstractAlfresc
|
||||
Date[] dates = new Date[5];
|
||||
|
||||
Calendar cal = new GregorianCalendar();
|
||||
cal.setTimeZone(TimeZone.getTimeZone("UTC"));
|
||||
for (int i = 0; i < dates.length; i++) {
|
||||
|
||||
cal.set(1980, i, 21);
|
||||
dates[i] = cal.getTime();
|
||||
}
|
||||
|
||||
-769
@@ -1,769 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8" ?>
|
||||
|
||||
<!--
|
||||
TODO
|
||||
- Geospatial ?
|
||||
- Currency ?
|
||||
- (default facetting configuration)
|
||||
-->
|
||||
<schema name="Alfresco V2.0" version="1.5">
|
||||
<types>
|
||||
<!-- ID and version to support SOLR cloud -->
|
||||
|
||||
<fieldType name="identifier" class="solr.StrField" sortMissingLast="true" positionIncrementGap="100" />
|
||||
<fieldType name="version" class="solr.TrieLongField" precisionStep="64" positionIncrementGap="0" />
|
||||
|
||||
|
||||
<fieldType name="lowercase_id" class="solr.TextField" sortMissingLast="true" omitNorms="true" positionIncrementGap="100" >
|
||||
<analyzer>
|
||||
<tokenizer class="solr.KeywordTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory" />
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Fixed analysis for numeric types - query, range, facet, and sorting-->
|
||||
|
||||
<fieldType name="int" class="solr.TrieIntField" precisionStep="4" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<fieldType name="float" class="solr.TrieFloatField" precisionStep="4" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<fieldType name="long" class="solr.TrieLongField" precisionStep="6" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<fieldType name="double" class="solr.TrieDoubleField" precisionStep="6" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<fieldType name="date" class="solr.TrieDateField" precisionStep="6" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
|
||||
<!-- Fixed stats analysis for numeric types - stats -->
|
||||
|
||||
<fieldType name="stats_int" class="solr.TrieIntField" precisionStep="32" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<fieldType name="stats_float" class="solr.TrieFloatField" precisionStep="64" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<fieldType name="stats_long" class="solr.TrieLongField" precisionStep="32" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<fieldType name="stats_double" class="solr.TrieDoubleField" precisionStep="64" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<fieldType name="stats_date" class="solr.TrieDateField" precisionStep="64" positionIncrementGap="100" sortMissingLast="false" sortMissingFirst="false" />
|
||||
|
||||
<!-- All text fields are handled via the Alfresco data model -->
|
||||
<!-- All anlysis is defined in dummy field types which this field type uses -->
|
||||
|
||||
<!-- TODO: truncated text field for sorting?? -->
|
||||
|
||||
<fieldType name="oldStandardAnalysis" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.ASCIIFoldingFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
|
||||
<fieldType name="alfrescoFieldType" class="org.alfresco.solr.AlfrescoFieldType" />
|
||||
|
||||
<fieldType name="alfrescoCollatableMLTextFieldType" class="org.alfresco.solr.AlfrescoCollatableMLTextFieldType" sortMissingLast="true" />
|
||||
|
||||
<fieldType name="path" class="solr.TextField" >
|
||||
<analyzer class="org.alfresco.repo.search.impl.lucene.analysis.PathAnalyzer" />
|
||||
</fieldType>
|
||||
|
||||
<fieldType name="location_rpt" class="solr.SpatialRecursivePrefixTreeFieldType"
|
||||
spatialContextFactory="org.locationtech.spatial4j.context.SpatialContextFactory"
|
||||
distErrPct="0.025"
|
||||
maxDistErr="0.000009"
|
||||
distanceUnits="degrees" />
|
||||
|
||||
|
||||
<!-- Suggestion -->
|
||||
|
||||
<fieldType name="text_shingle" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer type="index">
|
||||
<tokenizer class="solr.ICUTokenizerFactory"/>
|
||||
<filter class="solr.ICUNormalizer2FilterFactory" name="nfkc_cf" mode="compose" />
|
||||
<filter class="solr.ShingleFilterFactory" minShingleSize="2" maxShingleSize="3" outputUnigrams="true" outputUnigramsIfNoShingles="true" tokenSeparator=" " />
|
||||
</analyzer>
|
||||
<analyzer type="query">
|
||||
<tokenizer class="solr.ICUTokenizerFactory"/>
|
||||
<filter class="solr.ICUNormalizer2FilterFactory" name="nfkc_cf" mode="compose" />
|
||||
<filter class="solr.ShingleFilterFactory" minShingleSize="2" maxShingleSize="3" outputUnigrams="false" outputUnigramsIfNoShingles="true" tokenSeparator=" " />
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<fieldType name="text_min_hash" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer type="index">
|
||||
<tokenizer class="solr.ICUTokenizerFactory"/>
|
||||
<filter class="solr.ICUNormalizer2FilterFactory" name="nfkc_cf" mode="compose" />
|
||||
<filter class="solr.ShingleFilterFactory" minShingleSize="5" maxShingleSize="5" outputUnigrams="false" outputUnigramsIfNoShingles="false" tokenSeparator=" " />
|
||||
<!--
|
||||
<filter class="org.apache.lucene.analysis.minhash.ContextAccumulatingFilterFactory" />
|
||||
-->
|
||||
<filter class="org.apache.lucene.analysis.minhash.MinHashFilterFactory" hashCount="1" hashSetSize="1" bucketCount="512" />
|
||||
</analyzer>
|
||||
<analyzer type="query">
|
||||
<tokenizer class="solr.KeywordTokenizerFactory" />
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<fieldType name="text_plain" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.ICUTokenizerFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Cross-locale -->
|
||||
<!-- Include word splitting -->
|
||||
|
||||
<fieldType name="text___" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.ICUTokenizerFactory"/>
|
||||
<!-- <tokenizer class="org.apache.solr.analysis.WhitespaceTokenizerFactory" /> -->
|
||||
<filter class="org.apache.solr.analysis.WordDelimiterFilterFactory"
|
||||
generateWordParts="1"
|
||||
generateNumberParts="1"
|
||||
catenateWords="1"
|
||||
catenateNumbers="1"
|
||||
catenateAll="1"
|
||||
splitOnCaseChange="1"
|
||||
splitOnNumerics="1"
|
||||
preserveOriginal="1"
|
||||
stemEnglishPossessive="1"/>
|
||||
<filter class="solr.ICUFoldingFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
|
||||
<fieldType name="alfrescoCollatableTextFieldType" class="org.alfresco.solr.AlfrescoCollatableTextFieldType" sortMissingLast="true" />
|
||||
|
||||
<!-- -->
|
||||
<!-- Field types that are only used to define language specific analysis (generally ordered by ISO code) -->
|
||||
<!-- -->
|
||||
|
||||
<!-- English -->
|
||||
<fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer type="index">
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<!-- in this example, we will only use synonyms at query time
|
||||
<filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
|
||||
-->
|
||||
<filter class="solr.ICUFoldingFilterFactory"/>
|
||||
<filter class="solr.EnglishPossessiveFilterFactory"/>
|
||||
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
|
||||
<!-- Optionally you may want to use this less aggressive stemmer instead of PorterStemFilterFactory:
|
||||
<filter class="solr.EnglishMinimalStemFilterFactory"/>
|
||||
-->
|
||||
<filter class="solr.PorterStemFilterFactory"/>
|
||||
<filter class="solr.CommonGramsFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
|
||||
</analyzer>
|
||||
<analyzer type="query">
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
|
||||
<filter class="solr.ICUFoldingFilterFactory"/>
|
||||
<filter class="solr.EnglishPossessiveFilterFactory"/>
|
||||
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
|
||||
<!-- Optionally you may want to use this less aggressive stemmer instead of PorterStemFilterFactory:
|
||||
<filter class="solr.EnglishMinimalStemFilterFactory"/>
|
||||
-->
|
||||
<filter class="solr.PorterStemFilterFactory"/>
|
||||
<filter class="solr.CommonGramsQueryFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Arabic -->
|
||||
<fieldType name="text_ar" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<!-- for any non-arabic -->
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ar.txt" />
|
||||
<!-- normalizes ~ to ~, etc -->
|
||||
<filter class="solr.ArabicNormalizationFilterFactory"/>
|
||||
<filter class="solr.ArabicStemFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Bulgarian -->
|
||||
<fieldType name="text_bg" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_bg.txt" />
|
||||
<filter class="solr.BulgarianStemFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Catalan -->
|
||||
<fieldType name="text_ca" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<!-- removes l', etc -->
|
||||
<filter class="solr.ElisionFilterFactory" ignoreCase="true" articles="lang/contractions_ca.txt"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ca.txt" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Catalan"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- CJK bigram (see text_ja for a Japanese configuration using morphological analysis) -->
|
||||
<fieldType name="text_cjk" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<!-- normalize width before bigram, as e.g. half-width dakuten combine -->
|
||||
<filter class="solr.CJKWidthFilterFactory"/>
|
||||
<!-- for any non-CJK -->
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.CJKBigramFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Kurdish -->
|
||||
<fieldType name="text_ckb" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.SoraniNormalizationFilterFactory"/>
|
||||
<!-- for any latin text -->
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ckb.txt"/>
|
||||
<filter class="solr.SoraniStemFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Czech -->
|
||||
<fieldType name="text_cz" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_cz.txt" />
|
||||
<filter class="solr.CzechStemFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Danish -->
|
||||
<fieldType name="text_da" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_da.txt" format="snowball" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Danish"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- German -->
|
||||
<fieldType name="text_de" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_de.txt" format="snowball" />
|
||||
<filter class="solr.GermanNormalizationFilterFactory"/>
|
||||
<filter class="solr.GermanLightStemFilterFactory"/>
|
||||
<!-- less aggressive: <filter class="solr.GermanMinimalStemFilterFactory"/> -->
|
||||
<!-- more aggressive: <filter class="solr.SnowballPorterFilterFactory" language="German2"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Greek -->
|
||||
<fieldType name="text_el" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<!-- greek specific lowercase for sigma -->
|
||||
<filter class="solr.GreekLowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="false" words="lang/stopwords_el.txt" />
|
||||
<filter class="solr.GreekStemFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Spanish -->
|
||||
<fieldType name="text_es" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_es.txt" format="snowball" />
|
||||
<filter class="solr.SpanishLightStemFilterFactory"/>
|
||||
<!-- more aggressive: <filter class="solr.SnowballPorterFilterFactory" language="Spanish"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Basque -->
|
||||
<fieldType name="text_eu" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_eu.txt" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Basque"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Persian -->
|
||||
<fieldType name="text_fa" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<!-- for ZWNJ -->
|
||||
<charFilter class="solr.PersianCharFilterFactory"/>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.ArabicNormalizationFilterFactory"/>
|
||||
<filter class="solr.PersianNormalizationFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_fa.txt" />
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Finnish -->
|
||||
<fieldType name="text_fi" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_fi.txt" format="snowball" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Finnish"/>
|
||||
<!-- less aggressive: <filter class="solr.FinnishLightStemFilterFactory"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- French -->
|
||||
<fieldType name="text_fr" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<!-- removes l', etc -->
|
||||
<filter class="solr.ElisionFilterFactory" ignoreCase="true" articles="lang/contractions_fr.txt"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_fr.txt" format="snowball" />
|
||||
<filter class="solr.FrenchLightStemFilterFactory"/>
|
||||
<!-- less aggressive: <filter class="solr.FrenchMinimalStemFilterFactory"/> -->
|
||||
<!-- more aggressive: <filter class="solr.SnowballPorterFilterFactory" language="French"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Irish -->
|
||||
<fieldType name="text_ga" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<!-- removes d', etc -->
|
||||
<filter class="solr.ElisionFilterFactory" ignoreCase="true" articles="lang/contractions_ga.txt"/>
|
||||
<!-- removes n-, etc. position increments is intentionally false! -->
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/hyphenations_ga.txt"/>
|
||||
<filter class="solr.IrishLowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ga.txt"/>
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Irish"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Galician -->
|
||||
<fieldType name="text_gl" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_gl.txt" />
|
||||
<filter class="solr.GalicianStemFilterFactory"/>
|
||||
<!-- less aggressive: <filter class="solr.GalicianMinimalStemFilterFactory"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Hindi -->
|
||||
<fieldType name="text_hi" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<!-- normalizes unicode representation -->
|
||||
<filter class="solr.IndicNormalizationFilterFactory"/>
|
||||
<!-- normalizes variation in spelling -->
|
||||
<filter class="solr.HindiNormalizationFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_hi.txt" />
|
||||
<filter class="solr.HindiStemFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Hungarian -->
|
||||
<fieldType name="text_hu" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_hu.txt" format="snowball" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Hungarian"/>
|
||||
<!-- less aggressive: <filter class="solr.HungarianLightStemFilterFactory"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Armenian -->
|
||||
<fieldType name="text_hy" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_hy.txt" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Armenian"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Indonesian -->
|
||||
<fieldType name="text_id" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_id.txt" />
|
||||
<!-- for a less aggressive approach (only inflectional suffixes), set stemDerivational to false -->
|
||||
<filter class="solr.IndonesianStemFilterFactory" stemDerivational="true"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Italian -->
|
||||
<fieldType name="text_it" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<!-- removes l', etc -->
|
||||
<filter class="solr.ElisionFilterFactory" ignoreCase="true" articles="lang/contractions_it.txt"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_it.txt" format="snowball" />
|
||||
<filter class="solr.ItalianLightStemFilterFactory"/>
|
||||
<!-- more aggressive: <filter class="solr.SnowballPorterFilterFactory" language="Italian"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Japanese using morphological analysis (see text_cjk for a configuration using bigramming)
|
||||
|
||||
NOTE: If you want to optimize search for precision, use default operator AND in your query
|
||||
parser config with <solrQueryParser defaultOperator="AND"/> further down in this file. Use
|
||||
OR if you would like to optimize for recall (default).
|
||||
-->
|
||||
<fieldType name="text_ja" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="false">
|
||||
<analyzer>
|
||||
<!-- Kuromoji Japanese morphological analyzer/tokenizer (JapaneseTokenizer)
|
||||
|
||||
Kuromoji has a search mode (default) that does segmentation useful for search. A heuristic
|
||||
is used to segment compounds into its parts and the compound itself is kept as synonym.
|
||||
|
||||
Valid values for attribute mode are:
|
||||
normal: regular segmentation
|
||||
search: segmentation useful for search with synonyms compounds (default)
|
||||
extended: same as search mode, but unigrams unknown words (experimental)
|
||||
|
||||
For some applications it might be good to use search mode for indexing and normal mode for
|
||||
queries to reduce recall and prevent parts of compounds from being matched and highlighted.
|
||||
Use <analyzer type="index"> and <analyzer type="query"> for this and mode normal in query.
|
||||
|
||||
Kuromoji also has a convenient user dictionary feature that allows overriding the statistical
|
||||
model with your own entries for segmentation, part-of-speech tags and readings without a need
|
||||
to specify weights. Notice that user dictionaries have not been subject to extensive testing.
|
||||
|
||||
User dictionary attributes are:
|
||||
userDictionary: user dictionary filename
|
||||
userDictionaryEncoding: user dictionary encoding (default is UTF-8)
|
||||
|
||||
See lang/userdict_ja.txt for a sample user dictionary file.
|
||||
|
||||
Punctuation characters are discarded by default. Use discardPunctuation="false" to keep them.
|
||||
|
||||
See http://wiki.apache.org/solr/JapaneseLanguageSupport for more on Japanese language support.
|
||||
-->
|
||||
<tokenizer class="solr.JapaneseTokenizerFactory" mode="search"/>
|
||||
<!--<tokenizer class="solr.JapaneseTokenizerFactory" mode="search" userDictionary="lang/userdict_ja.txt"/>-->
|
||||
<!-- Reduces inflected verbs and adjectives to their base/dictionary forms (~~~) -->
|
||||
<filter class="solr.JapaneseBaseFormFilterFactory"/>
|
||||
<!-- Removes tokens with certain part-of-speech tags -->
|
||||
<filter class="solr.JapanesePartOfSpeechStopFilterFactory" tags="lang/stoptags_ja.txt" />
|
||||
<!-- Normalizes full-width romaji to half-width and half-width kana to full-width (Unicode NFKC subset) -->
|
||||
<filter class="solr.CJKWidthFilterFactory"/>
|
||||
<!-- Removes common tokens typically not useful for search, but have a negative effect on ranking -->
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ja.txt" />
|
||||
<!-- Normalizes common katakana spelling variations by removing any last long sound character (U+30FC) -->
|
||||
<filter class="solr.JapaneseKatakanaStemFilterFactory" minimumLength="4"/>
|
||||
<!-- Lower-cases romaji characters -->
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Latvian -->
|
||||
<fieldType name="text_lv" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_lv.txt" />
|
||||
<filter class="solr.LatvianStemFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Dutch -->
|
||||
<fieldType name="text_nl" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_nl.txt" format="snowball" />
|
||||
<filter class="solr.StemmerOverrideFilterFactory" dictionary="lang/stemdict_nl.txt" ignoreCase="false"/>
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Dutch"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Norwegian -->
|
||||
<fieldType name="text_no" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_no.txt" format="snowball" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Norwegian"/>
|
||||
<!-- less aggressive: <filter class="solr.NorwegianLightStemFilterFactory" variant="nb"/> -->
|
||||
<!-- singular/plural: <filter class="solr.NorwegianMinimalStemFilterFactory" variant="nb"/> -->
|
||||
<!-- The "light" and "minimal" stemmers support variants: nb=Bokml, nn=Nynorsk, no=Both -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Portuguese -->
|
||||
<fieldType name="text_pt" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_pt.txt" format="snowball" />
|
||||
<filter class="solr.PortugueseLightStemFilterFactory"/>
|
||||
<!-- less aggressive: <filter class="solr.PortugueseMinimalStemFilterFactory"/> -->
|
||||
<!-- more aggressive: <filter class="solr.SnowballPorterFilterFactory" language="Portuguese"/> -->
|
||||
<!-- most aggressive: <filter class="solr.PortugueseStemFilterFactory"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Romanian -->
|
||||
<fieldType name="text_ro" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ro.txt" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Romanian"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Russian -->
|
||||
<fieldType name="text_ru" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ru.txt" format="snowball" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Russian"/>
|
||||
<!-- less aggressive: <filter class="solr.RussianLightStemFilterFactory"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Swedish -->
|
||||
<fieldType name="text_sv" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_sv.txt" format="snowball" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Swedish"/>
|
||||
<!-- less aggressive: <filter class="solr.SwedishLightStemFilterFactory"/> -->
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Thai -->
|
||||
<fieldType name="text_th" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="org.apache.lucene.analysis.th.ThaiTokenizerFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_th.txt" />
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
<!-- Turkish -->
|
||||
<fieldType name="text_tr" class="solr.TextField" positionIncrementGap="100">
|
||||
<analyzer>
|
||||
<tokenizer class="solr.StandardTokenizerFactory"/>
|
||||
<!-- <filter class="solr.ApostropheFilterFactory"/> -->
|
||||
<filter class="solr.TurkishLowerCaseFilterFactory"/>
|
||||
<filter class="solr.StopFilterFactory" ignoreCase="false" words="lang/stopwords_tr.txt" />
|
||||
<filter class="solr.SnowballPorterFilterFactory" language="Turkish"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
|
||||
|
||||
</types>
|
||||
<fields>
|
||||
<!-- For SOLR cloud - should be the node version -->
|
||||
<field name="_version_" type="version" indexed="false" stored="true" docValues="true" required="true" />
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- For block join - currently not used -->
|
||||
<field name="_root_" type="identifier" indexed="true" stored="false"/>
|
||||
|
||||
<!-- Unique identifier - based on DBID -->
|
||||
<field name="id" type="identifier" indexed="true" omitNorms="true" stored="true" multiValued="false" required="true" docValues="true"/>
|
||||
<field name="MINHASH" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" required="false" docValues="false"/>
|
||||
|
||||
<!-- Special fields -->
|
||||
<field name="LID" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<field name="PARENT" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<field name="PATH" type="path" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<field name="ANCESTOR" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<field name="QNAME" type="path" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<field name="PRIMARYASSOCQNAME" type="path" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<field name="PRIMARYASSOCTYPEQNAME" type="path" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<field name="ISNODE" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<field name="ASSOCTYPEQNAME" type="path" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<field name="PRIMARYPARENT" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<field name="TYPE" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="true" />
|
||||
<field name="ASPECT" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="PROPERTIES" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="NULLPROPERTIES" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="FIELDS" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="FTSSTATUS" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<field name="DBID" type="long" indexed="true" omitNorms="true" stored="true" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="TXID" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="INTXID" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="ACLTXID" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="INACLTXID" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="TXCOMMITTIME" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="ACLTXCOMMITTIME" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="EXCEPTIONMESSAGE" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<field name="EXCEPTIONSTACK" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<field name="ACLID" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="READER" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="DENIED" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="OWNER" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="PARENTASSOCCRC" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="TENANT" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="true" />
|
||||
<field name="CASCADETX" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
|
||||
<!-- Used to store the last transaction and acl transaction for real time get -->
|
||||
<field name="S_TXID" type="long" indexed="true" omitNorms="true" stored="true" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="S_INTXID" type="long" indexed="true" omitNorms="true" stored="true" multiValued="false" sortMissingLast="false" sortMissingFirst="false"/>
|
||||
<field name="S_ACLTXID" type="long" indexed="true" omitNorms="true" stored="true" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="S_INACLTXID" type="long" indexed="true" omitNorms="true" stored="true" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="S_TXCOMMITTIME" type="long" indexed="true" omitNorms="true" stored="true" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<field name="S_ACLTXCOMMITTIME" type="long" indexed="true" omitNorms="true" stored="true" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
|
||||
<!-- SOLR doc type -->
|
||||
<field name="DOC_TYPE" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
|
||||
<!-- GEO -->
|
||||
<field name="GEO" type="location_rpt" indexed="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
|
||||
|
||||
|
||||
<!-- Suggestion -->
|
||||
<field name="suggest" type="text_shingle" indexed="true" omitNorms="true" omitPositions="true" stored="false" multiValued="true" />
|
||||
<field name="min_hash" type="text_min_hash" indexed="true" omitNorms="true" omitPositions="true" stored="false" multiValued="true" />
|
||||
|
||||
<!-- Dummy field used to create no matches for unknonw properties -->
|
||||
<field name="_dummy_" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
|
||||
|
||||
<!-- example pluggable field -->
|
||||
<field name="SITE" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="TAG" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="PNAME" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="NPATH" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="ANAME" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<field name="APATH" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Single value indexed -->
|
||||
<dynamicField name="any@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="encrypted@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="int@s_@*" type="int" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="long@s_@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="float@s_@*" type="float" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="double@s_@*" type="double" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="date@s_@*" type="date" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="datetime@s_@*" type="date" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="boolean@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="qname@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="category@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="noderef@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="noderef@sd_@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" sortMissingLast="true" />
|
||||
<dynamicField name="childassocref@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="assocref@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="path@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="locale@s_@*" type="lowercase_id" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="period@s_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
|
||||
<!-- Single + doc values value wildcard matches -->
|
||||
<dynamicField name="int@sd@*" type="int" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="long@sd@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="float@sd@*" type="float" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="double@sd@*" type="double" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="date@sd@*" type="date" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="datetime@sd@*" type="date" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
|
||||
|
||||
<!-- Multi - value indexed -->
|
||||
<dynamicField name="any@m_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="encrypted@m_@*" type="alfrescoFieldType" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="int@m_@*" type="int" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="long@m_@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="float@m_@*" type="float" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="double@m_@*" type="double" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="date@m_@*" type="date" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="datetime@m_@*" type="date" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="boolean@m_@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="boolean@md_@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" sortMissingLast="true"/>
|
||||
<dynamicField name="qname@m_@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="qname@md_@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" sortMissingLast="true"/>
|
||||
<dynamicField name="category@m_@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="category@md_@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" sortMissingLast="true" />
|
||||
<dynamicField name="category@sd_@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="true" />
|
||||
<dynamicField name="noderef@m_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="noderef@md@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" sortMissingLast="true" />
|
||||
<dynamicField name="childassocref@m_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="assocref@m_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="path@m_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="locale@m_@*" type="lowercase_id" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="period@m_@*" type="oldStandardAnalysis" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
|
||||
<!-- Multi + doc values value wildcard matches -->
|
||||
<dynamicField name="int@md@*" type="int" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="long@md@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="float@md@*" type="float" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="double@md@*" type="double" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="date@md@*" type="date" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="datetime@md@*" type="date" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
|
||||
|
||||
<dynamicField name="text@s____@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="text@s__l_@*" type="alfrescoFieldType" indexed="true" omitNorms="true" stored="false" multiValued="false" />
|
||||
<dynamicField name="text@s__lt@*" type="alfrescoFieldType" indexed="true" omitNorms="false" stored="false" multiValued="false" />
|
||||
<dynamicField name="text@s___t@*" type="text___" indexed="true" omitNorms="false" stored="false" multiValued="false" />
|
||||
<dynamicField name="text@s__sort@*" type="alfrescoCollatableTextFieldType" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
|
||||
<dynamicField name="text@sd___@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="true" />
|
||||
|
||||
<dynamicField name="text@m____@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="text@m__l_@*" type="alfrescoFieldType" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="text@m__lt@*" type="alfrescoFieldType" indexed="true" omitNorms="false" stored="false" multiValued="true" />
|
||||
<dynamicField name="text@m___t@*" type="text___" indexed="true" omitNorms="false" stored="false" multiValued="true" />
|
||||
|
||||
<dynamicField name="text@md___@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
|
||||
<dynamicField name="mltext@m____@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="mltext@m__l_@*" type="alfrescoFieldType" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="mltext@m__lt@*" type="alfrescoFieldType" indexed="true" omitNorms="false" stored="false" multiValued="true" />
|
||||
<dynamicField name="mltext@m___t@*" type="text___" indexed="true" omitNorms="false" stored="false" multiValued="true" />
|
||||
<dynamicField name="mltext@m__sort@*" type="alfrescoCollatableMLTextFieldType" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
|
||||
<dynamicField name="content@s__size@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="content@s__locale@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="true" />
|
||||
<dynamicField name="content@s__mimetype@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="true" />
|
||||
<dynamicField name="content@s__encoding@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="true" />
|
||||
<dynamicField name="content@s__docid@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" docValues="true" sortMissingLast="false" sortMissingFirst="false" />
|
||||
<dynamicField name="content@s__tr_ex@*" type="text___" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="content@s__tr_time@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
<dynamicField name="content@s__tr_status@*" type="lowercase_id" indexed="true" omitNorms="true" stored="false" multiValued="false" sortMissingLast="true" />
|
||||
|
||||
|
||||
<dynamicField name="content@s____@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="false" termPositions="false" />
|
||||
<dynamicField name="content@s__l_@*" type="alfrescoFieldType" indexed="true" omitNorms="true" stored="false" multiValued="false" termPositions="false" />
|
||||
<dynamicField name="content@s__lt@*" type="alfrescoFieldType" indexed="true" omitNorms="false" stored="false" multiValued="false" />
|
||||
<dynamicField name="content@s___t@*" type="text___" indexed="true" omitNorms="false" stored="false" multiValued="false" />
|
||||
|
||||
|
||||
<dynamicField name="content@m__size@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="content@m__locale@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="content@m__mimetype@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="content@m__encoding@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="content@m__docid@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="true" docValues="true" />
|
||||
<dynamicField name="content@m__tr_ex@*" type="text___" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="content@m__tr_time@*" type="long" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
<dynamicField name="content@m__tr_status@*" type="lowercase_id" indexed="true" omitNorms="true" stored="false" multiValued="true" />
|
||||
|
||||
|
||||
<dynamicField name="content@m____@*" type="identifier" indexed="true" omitNorms="true" stored="false" multiValued="true" termPositions="false" />
|
||||
<dynamicField name="content@m__l_@*" type="alfrescoFieldType" indexed="true" omitNorms="true" stored="false" multiValued="true" termPositions="false" />
|
||||
<dynamicField name="content@m__lt@*" type="alfrescoFieldType" indexed="true" omitNorms="false" stored="false" multiValued="true" />
|
||||
<dynamicField name="content@m___t@*" type="text___" indexed="true" omitNorms="false" stored="false" multiValued="true" />
|
||||
|
||||
|
||||
<!-- Suggestion -->
|
||||
<dynamicField name="suggest_*" type="text_shingle" indexed="false" omitNorms="true" stored="false" multiValued="true" />
|
||||
|
||||
<copyField source="suggest_*" dest="suggest" />
|
||||
|
||||
</fields>
|
||||
|
||||
<uniqueKey>id</uniqueKey>
|
||||
|
||||
</schema>
|
||||
Reference in New Issue
Block a user