diff --git a/search-services/alfresco-search/doc/architecture/decisions/0003-combined-codebase.md b/search-services/alfresco-search/doc/architecture/decisions/0003-combined-codebase.md new file mode 100644 index 000000000..6f7bc45be --- /dev/null +++ b/search-services/alfresco-search/doc/architecture/decisions/0003-combined-codebase.md @@ -0,0 +1,46 @@ +# 3. Combined Codebase + +Date: 09/04/2019 + +## Status + +Approved + +## Context + +Historically Alfresco has had a bad experience of having a monolithic codebase in SVN. The main issue with this was the +inability to easily work on feature branches, resulting in frequent conflicting changes. To make matters worse this was +seen as hard to move to git, since GitHub has/had a maximum limit on repository size and there were several large +binary artifacts stored in the Alfresco history [[1]]. + +More recently the Alfresco codebase has been split into a large number of small git repositories [[2]]. + +This proposal is to merge the various git repositories together. By doing this we aim to: + +1. Remove the effort of creating releases of library projects. +1. Allow tests to be written on the same branch (in the same repository) as the production code (i.e. facilitate TDD). +1. Make it easier for everyone (especially people outside the team and new starters) to find the 'right bit' of code. + +[1]: https://community.alfresco.com/community/ecm/blog/2015/04/01/so-when-is-alfresco-moving-to-github + +[2]: https://ts.alfresco.com/share/proxy/alfresco/api/node/content/versionStore/version2Store/a0c2492f-6354-4b98-adfc-e63d5c2209f5/SearchCodeBase.png + +## Decision + +We will merge the search-related repositories together and preserve their history. We will not attempt to merge code +that other teams also need (for example the TAS test utilities or the alfresco-data-model projects). + +## Consequences + +An epic has been raised containing the next steps [[3]]. Broadly speaking these are: + +1. Merge the alfresco-solr-client library with the SearchServices repository. +1. Merge the insight-engine, insight-jdbc and insight-zeppelin repositories. + 1. A consequence of this is that future releases of these artifacts will share version numbers. +1. Put all the end-to-end tests in a single testing project. +1. Merge the (public) SearchServices code into the (private) InsightEngine repository and set up a mirror so the +community code is still publicly editable. +1. Merge the tests with the production code. +1. Merge the build scripts with the production code. + +[3]: https://issues.alfresco.com/jira/browse/SEARCH-1393 diff --git a/search-services/alfresco-search/doc/architecture/decisions/0004-community-mirror.md b/search-services/alfresco-search/doc/architecture/decisions/0004-community-mirror.md new file mode 100644 index 000000000..f94890df2 --- /dev/null +++ b/search-services/alfresco-search/doc/architecture/decisions/0004-community-mirror.md @@ -0,0 +1,44 @@ +# 4. Community Mirror + +Date: 09/04/2019 + +## Status + +Approved + +## Context + +In [ADR 3: "Combined Codebase"](0003-combined-codebase.md) we decided to merge the Search Services and Insight Engine +repositories. Since we want to enable the community to submit pull requests to the Search Services project we need a +way to keep this code up to date on GitHub. + +## Decision + +We will mirror `master` and all branches starting with `release/` to a branch with the same name on GitHub. We will +exclude the alfresco-insight-engine-parent directory. We will include these commands as part of our build to do this: + +``` +# This avoids making changes to the original branch. +get checkout -b tempBranch +# This strips all enterprise changes (in a reproducible way) and pushes any updates to the mirror. +git filter-branch -f --prune-empty --index-filter 'git rm -r --cached --ignore-unmatch alfresco-insight-engine-parent' +git push out HEAD:$branch +# This resets us back to where we were before the filtering. +git checkout $branch +``` + +## Consequences + +A ticket has been raised to track this [[1]]. A script has been written to do the initial codebase merge and +mirroring [[2]]. + +We will rewrite the whole history of the SearchServices repository using the mirroring command. This will result in the +new repository containing a new root pom file and the search services code beneath that. + +All changes except those within the module called insight-engine will be mirrored. + +The root pom file in the community project will include a reference to the insight-engine pom file, so community users +will have to build from within the alfresco-search-parent directory. + +[1]: https://issues.alfresco.com/jira/browse/SEARCH-1397 +[2]: https://git.alfresco.com/search_discovery/combinerScript/blob/master/combineSearch.sh diff --git a/search-services/alfresco-search/doc/architecture/decisions/0005-merge-tests-and-production-code.md b/search-services/alfresco-search/doc/architecture/decisions/0005-merge-tests-and-production-code.md new file mode 100644 index 000000000..8af97eb65 --- /dev/null +++ b/search-services/alfresco-search/doc/architecture/decisions/0005-merge-tests-and-production-code.md @@ -0,0 +1,25 @@ +# 5. Merge Tests And Production Code + +Date: 09/04/2019 + +## Status + +Approved + +## Context + +In [ADR 3: "Combined Codebase"](0003-combined-codebase.md) we decided to merge the production and end-to-end test +repositories. In [ADR 4: "Community Mirror"](0004-community-mirror.md) we discuss setting up a mirror for the community +code. + +## Decision + +We will separate the end-to-end test code in half so that any code solely related to Insight Engine won't be mirrored. + +We will remove the existing test groups for the different versions of Search Services and Insight Engine, and instead +delete any tests from branches where they should not be run. + +## Consequences + +It will be possible to include production code changes along with all required test changes in the same merge request. +It will be easy to get new tests running before production code is written without causing other branches to fail. diff --git a/search-services/alfresco-search/doc/architecture/decisions/0006-release-synchronisation.md b/search-services/alfresco-search/doc/architecture/decisions/0006-release-synchronisation.md new file mode 100644 index 000000000..b0db4e15d --- /dev/null +++ b/search-services/alfresco-search/doc/architecture/decisions/0006-release-synchronisation.md @@ -0,0 +1,47 @@ +# 6. Release Synchronisation + +Date: 24/04/2019 + +## Status + +Approved + +## Context + +In [ADR 3: "Combined Codebase"](0003-combined-codebase.md) we decided to merge the Search Services and Insight Engine +repositories. In [ADR 4: "Community Mirror"](0004-community-mirror.md) we discussed how we would set up a build job to +ensure community code is available to the community. In particular we decided: + +> We will mirror `master` and all branches starting with `release/` to a branch with the same name on GitHub. + +## Decision + +During the combining of the Search Services and Insight Engine codebases we will create a branch `master` by merging +the existing `master` branches of those two projects. We will update the version of Insight Engine to match Search Services - that +is 1.4.0-SNAPSHOT. + +## Consequences + +The next minor release of Insight Engine will be 1.4.0 which will happen at the same time as the release of Search +Services 1.4.0. There will be no Insight Engine releases with the versions 1.2.x or 1.3.x. + +Here are a few examples to illustrate the process for future releases. Note that other releases (including service +packs, etc.) will follow a similar logic based on whether they are pre- or post- 1.4.0. + +* SS 1.3.0 needs a hotfix + +We release from the `release/alfresco-search-services/V1.3.0.x` branch. + +* IE 1.1.0 needs a hotfix + +We release from the `release/alfresco-insight-engine/V1.1.0.x` branch. We need to use the maven version 1.1.0.2 for the +IE modules and 1.4.0-IE1.1.0.2 for the SS modules. + +* SS 1.4.0 needs a hotfix + +We'll create a hotfix release for both SS and IE (since they will both have the same issue). This will be from the +`release/V1.4.0.x` branch (which would have already been created if SS/IE 1.4.0 had been released). + +* IE 1.4.0 needs a hotfix + +We'll create a hotfix release for both SS and IE from `release/V1.4.0.x` (and not publicise SS 1.4.0.1). diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java index 6d38cd5d8..c4287b22a 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java @@ -145,6 +145,7 @@ import org.apache.lucene.util.BytesRefBuilder; import org.apache.solr.common.SolrDocument; import org.apache.solr.common.SolrDocumentList; import org.apache.solr.common.SolrInputDocument; +import org.apache.solr.common.SolrInputField; import org.apache.solr.common.params.CommonParams; import org.apache.solr.common.params.FacetParams; import org.apache.solr.common.params.ModifiableSolrParams; @@ -2354,6 +2355,10 @@ public class SolrInformationServer implements InformationServer if (cachedDoc != null) { + ofNullable(cachedDoc.getField("MINHASH")) + .map(SolrInputField::getValue) + .ifPresent(minHash -> newDoc.setField("MINHASH", minHash)); + // Builds up the new solr doc from the cached content regardless of whether or not it is current List fields = AlfrescoSolrDataModel.getInstance().getIndexedFieldNamesForProperty( propertyQName).getFields(); diff --git a/search-services/alfresco-search/src/test/java/org/alfresco/solr/query/AlfrescoSolrFingerprintTest.java b/search-services/alfresco-search/src/test/java/org/alfresco/solr/query/AlfrescoSolrFingerprintTest.java index 01fde6786..96e1285c0 100644 --- a/search-services/alfresco-search/src/test/java/org/alfresco/solr/query/AlfrescoSolrFingerprintTest.java +++ b/search-services/alfresco-search/src/test/java/org/alfresco/solr/query/AlfrescoSolrFingerprintTest.java @@ -19,50 +19,75 @@ package org.alfresco.solr.query; +import org.alfresco.model.ContentModel; import org.alfresco.repo.search.adaptor.lucene.QueryConstants; import org.alfresco.solr.AbstractAlfrescoSolrTests; -import org.alfresco.solr.client.*; -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; +import org.alfresco.solr.client.Acl; +import org.alfresco.solr.client.AclChangeSet; +import org.alfresco.solr.client.AclReaders; +import org.alfresco.solr.client.Node; +import org.alfresco.solr.client.NodeMetaData; +import org.alfresco.solr.client.SOLRAPIQueueClient; +import org.alfresco.solr.client.StringPropertyValue; +import org.alfresco.solr.client.Transaction; import org.apache.lucene.index.Term; import org.apache.lucene.search.BooleanClause; import org.apache.lucene.search.BooleanQuery; import org.apache.lucene.search.LegacyNumericRangeQuery; import org.apache.lucene.search.TermQuery; -import org.apache.lucene.util.LuceneTestCase; import org.apache.solr.common.params.ModifiableSolrParams; import org.junit.After; import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; -import java.util.ArrayList; import java.util.List; import java.util.Random; +import java.util.stream.Collectors; +import static java.util.Arrays.asList; +import static java.util.Arrays.stream; +import static java.util.Collections.singletonList; +import static java.util.stream.IntStream.range; import static org.alfresco.solr.AlfrescoSolrUtils.*; -@LuceneTestCase.SuppressCodecs({"Appending","Lucene3x","Lucene40","Lucene41","Lucene42","Lucene43", "Lucene44", "Lucene45","Lucene46","Lucene47","Lucene48","Lucene49"}) public class AlfrescoSolrFingerprintTest extends AbstractAlfrescoSolrTests { - private static Log logger = LogFactory.getLog(AlfrescoSolrFingerprintTest.class); private static long MAX_WAIT_TIME = 80000; + @BeforeClass public static void beforeClass() throws Exception { - initAlfrescoCore("schema-fingerprint.xml"); + initAlfrescoCore("schema.xml"); } + private Acl acl; + @Before - public void setUp() throws Exception { - // if you override setUp or tearDown, you better callf - // the super classes version - //clearIndex(); - //assertU(commit()); + public void prepare() throws Exception + { + AclChangeSet aclChangeSet = getAclChangeSet(1); + acl = getAcl(aclChangeSet); + Acl acl2 = getAcl(aclChangeSet); + + AclReaders aclReaders = getAclReaders(aclChangeSet, acl, singletonList("joel"), singletonList("phil"), null); + AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, singletonList("jim"), singletonList("phil"), null); + + indexAclChangeSet(aclChangeSet, + asList(acl, acl2), + asList(aclReaders, aclReaders2)); + + // Check for the ACL state stamp. + BooleanQuery.Builder builder = new BooleanQuery.Builder(); + builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!ACLTX")), BooleanClause.Occur.MUST)); + builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_ACLTXID, aclChangeSet.getId(), aclChangeSet.getId() + 1, true, false), BooleanClause.Occur.MUST)); + BooleanQuery waitForQuery = builder.build(); + waitForDocCount(waitForQuery, 1, MAX_WAIT_TIME); } @After - public void clearQueue() throws Exception { + public void clearQueue() + { SOLRAPIQueueClient.nodeMetaDataMap.clear(); SOLRAPIQueueClient.transactionQueue.clear(); SOLRAPIQueueClient.aclChangeSetQueue.clear(); @@ -70,234 +95,255 @@ public class AlfrescoSolrFingerprintTest extends AbstractAlfrescoSolrTests SOLRAPIQueueClient.aclMap.clear(); SOLRAPIQueueClient.nodeMap.clear(); SOLRAPIQueueClient.nodeContentMap.clear(); + + clearIndex(); + assertU(commit()); } - - @Test - public void testBasciFingerPrint() throws Exception + private void makeSureTransactionHasBeenIndexed(long transactionId) throws Exception { - /* - * Create and index an AclChangeSet. - */ - - logger.info("######### Starting fingerprint test ###########"); - AclChangeSet aclChangeSet = getAclChangeSet(1); - - Acl acl = getAcl(aclChangeSet); - Acl acl2 = getAcl(aclChangeSet); - - - AclReaders aclReaders = getAclReaders(aclChangeSet, acl, list("joel"), list("phil"), null); - AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, list("jim"), list("phil"), null); - - - indexAclChangeSet(aclChangeSet, - list(acl, acl2), - list(aclReaders, aclReaders2)); - - - //Check for the ACL state stamp. + //Check for the TXN state stamp. BooleanQuery.Builder builder = new BooleanQuery.Builder(); - builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!ACLTX")), BooleanClause.Occur.MUST)); - builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_ACLTXID, aclChangeSet.getId(), aclChangeSet.getId() + 1, true, false), BooleanClause.Occur.MUST)); + builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!TX")), BooleanClause.Occur.MUST)); + builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_TXID, transactionId, transactionId + 1, true, false), BooleanClause.Occur.MUST)); BooleanQuery waitForQuery = builder.build(); waitForDocCount(waitForQuery, 1, MAX_WAIT_TIME); + } - logger.info("#################### Passed First Test ##############################"); - - /* - * Create and index a Transaction - */ - - //First create a transaction. + @Test + public void testBasicFingerprint() throws Exception + { Transaction txn = getTransaction(0, 4); - //Next create two nodes to update for the transaction Node node1 = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED); Node node2 = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED); Node node3 = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED); Node node4 = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED); - - //Next create the NodeMetaData for each node. TODO: Add more metadata NodeMetaData nodeMetaData1 = getNodeMetaData(node1, txn, acl, "mike", null, false); NodeMetaData nodeMetaData2 = getNodeMetaData(node2, txn, acl, "mike", null, false); NodeMetaData nodeMetaData3 = getNodeMetaData(node3, txn, acl, "mike", null, false); NodeMetaData nodeMetaData4 = getNodeMetaData(node4, txn, acl, "mike", null, false); - List content = new ArrayList(); + Random randomizer = new Random(1); + String aFirstToken = Integer.toString(Math.abs(randomizer.nextInt())); + + indexTransaction(txn, + asList(node1, node2, node3, node4), + asList(nodeMetaData1, nodeMetaData2, nodeMetaData3, nodeMetaData4), + randomTextContent()); + + makeSureTransactionHasBeenIndexed(txn.getId()); + makeSureContentNodesHaveBeenIndexed("mike", aFirstToken, 4); + + assertFingerprintQueryCorrectness(node1.getId(), + "*[count(//doc)= 4]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", + "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']", + "//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']"); + + assertFingerprintQueryCorrectness(nodeMetaData1.getNodeRef().getId(), + "*[count(//doc)= 4]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", + "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']", + "//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']"); + + assertFingerprintQueryCorrectness(node1.getId() + "_70", + "*[count(//doc)= 2]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']"); + + assertFingerprintQueryCorrectness(nodeMetaData1.getNodeRef().getId() + "_70", + "*[count(//doc)= 2]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']"); + + assertFingerprintQueryCorrectness(node1.getId() + "_45", + "*[count(//doc)= 3]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", + "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']"); + + assertFingerprintQueryCorrectness(nodeMetaData1.getNodeRef().getId() + "_45", + "*[count(//doc)= 3]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", + "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']"); + + assertFingerprintQueryCorrectness(node4.getId() + "_30", + "*[count(//doc)= 4]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", + "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']", + "//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']"); + + assertFingerprintQueryCorrectness(nodeMetaData4.getNodeRef().getId() + "_30", + "*[count(//doc)= 4]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", + "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']", + "//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']"); + + assertFingerprintQueryCorrectness(node4.getId(), + "*[count(//doc)= 4]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", + "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']", + "//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']"); + + assertFingerprintQueryCorrectness(nodeMetaData4.getNodeRef().getId(), + "*[count(//doc)= 4]", + "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", + "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", + "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']", + "//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']"); + } + + @Test + public void testFingerprintStillExistsAfterNodeMetadataUpdate() throws Exception + { + Transaction txn = getTransaction(0, 1); + Node fileNode = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED); + NodeMetaData fileMetaData = getNodeMetaData(fileNode, txn, acl, "mike", null, false); + + indexTransaction( + txn, + singletonList(fileNode), + singletonList(fileMetaData), + singletonList("This is a text content which is longer than the default hello world " + fileNode.getId() + + " returned by the Mock SOLRAPIQueueClient. This is needed because the \"min_hash\" field type " + + "definition in Solr doesn't take in account fields which produce less than 5 tokens (see the " + + "ShingleFilter settings).")); + + makeSureTransactionHasBeenIndexed(txn.getId()); + makeSureContentNodeHasBeenIndexed(fileNode, "mike", "world"); + + assertFingerprintQueryCorrectness(fileNode.getId(), "*[count(//doc)=1]","//result/doc[1]/long[@name='DBID'][.='" + fileNode.getId() + "']"); + + // Let's update the test node + fileMetaData.setOwner("Andrea"); + fileMetaData.getProperties().put(ContentModel.PROP_TITLE, new StringPropertyValue("This is the new file \"title\" metadata attribute.")); + reindexTransactionId(txn.getId()); + + makeSureContentNodeHasBeenIndexed(fileNode, "Andrea", "world"); + + assertFingerprintQueryCorrectness(fileNode.getId(), "*[count(//doc)=1]","//result/doc[1]/long[@name='DBID'][.='" + fileNode.getId() + "']"); + } + + /** + * Queries the index using a token from the (dummy) text produced by the test framework ("world", actually). + * Once the query returns a positive result we are sure the ContentTracker + * + *
    + *
  1. + * Fetched the text content associated with the current node, from Alfresco + *
    2. + *
  2. + * Computed a fingerprint (using the retrieved text) for the node + *
    3. + *
  3. + * Updated the node definition in the (Solr)ContentStore and in Solr + *
    4. + *
+ * + * Last but not least, we are also making sure that CommitTracker executed its cycle as well (otherwise document + * wouldn't be searchable). + * + * @param node an addition term which will be appended as a required clause in the executed query. + * @param testTerm a term which is supposed to be in the indexed content + * @param owner the #FIELD_OWNER which will be used as an additional required query clause. + * @throws Exception in case the MAX_WAIT_TIME is reached and the node is not in results. + */ + private void makeSureContentNodeHasBeenIndexed(final Node node, final String owner, String testTerm) throws Exception + { + waitForDocCount(new TermQuery(new Term(QueryConstants.FIELD_READER, "jim")), 1, MAX_WAIT_TIME); + waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), 1, MAX_WAIT_TIME); + + waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", Long.toString(node.getId()))), 1, MAX_WAIT_TIME); + + BooleanQuery.Builder builder = new BooleanQuery.Builder(); + builder.add(new BooleanClause(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), BooleanClause.Occur.MUST)); + builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_OWNER, owner)), BooleanClause.Occur.MUST)); + waitForDocCount(builder.build(), 1, MAX_WAIT_TIME); + } + + + /** + * Queries the index using a token from the (dummy) text produced by the test framework. + * Once the query returns a positive result we are sure the ContentTracker + * + *
    + *
  1. + * Fetched the text content associated with the test nodes, from Alfresco + *
    2. + *
  2. + * Computed a fingerprint (using the retrieved text) for each node + *
    3. + *
  3. + * Updated the nodes definitions in the (Solr)ContentStore and in Solr + *
    4. + *
+ * + * Last but not least, we are also making sure that CommitTracker executed its cycle as well (otherwise documents + * wouldn't be searchable). + * + * @param owner the #FIELD_OWNER which will be used as an additional required query clause. + * @param testTerm a term which is supposed to be in the indexed content + * @throws Exception in case the MAX_WAIT_TIME is reached and the node is not in results. + */ + private void makeSureContentNodesHaveBeenIndexed(final String owner, String testTerm, final int expectedCount) throws Exception + { + waitForDocCount(new TermQuery(new Term(QueryConstants.FIELD_READER, "jim")), 1, MAX_WAIT_TIME); + waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), expectedCount, MAX_WAIT_TIME); + + BooleanQuery.Builder builder = new BooleanQuery.Builder(); + builder.add(new BooleanClause(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), BooleanClause.Occur.MUST)); + builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_OWNER, owner)), BooleanClause.Occur.MUST)); + waitForDocCount(builder.build(), expectedCount, MAX_WAIT_TIME); + } + + /** + * Asserts that a query in the format 
FINGERPRINT:
correctly returns the node we are testing. + * + * @param id the node identifier. + */ + private void assertFingerprintQueryCorrectness(long id, String ... assertions) + { + assertFingerprintQueryCorrectness(String.valueOf(id), assertions); + } + + /** + * Asserts that a query in the format 
FINGERPRINT:
correctly returns the node we are testing. + * + * @param id the node identifier. + */ + private void assertFingerprintQueryCorrectness(String id, String ... assertions) + { + ModifiableSolrParams params = new ModifiableSolrParams() + .add("q", "FINGERPRINT:" + id) + .add("qt", "/afts") + .add("start", "0") + .add("rows", "6") + .add("sort", "id asc"); + // .add("fq", "{!afts}AUTHORITY_FILTER_FROM_JSON"); + + SolrServletRequest req = areq(params, "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [ \"joel\"], \"tenants\": [ \"\" ]}"); + assertQ(req, assertions); + } + + private List randomTextContent() + { int[] sizes = {2000, 1000, 1500, 750}; - Random r = new Random(1); - String token1 = Integer.toString(Math.abs(r.nextInt())); - - for(int i=0; i<4; i++) { - Random rand = new Random(1); - StringBuilder buf = new StringBuilder(); - int size = sizes[i]; - for(int s=0; s0) { - buf.append(" "); - } - buf.append(Integer.toString(Math.abs(rand.nextInt()))); - } - content.add(buf.toString()); - } - - //Index the transaction, nodes, and nodeMetaDatas. - //Note that the content is automatically created by the test framework. - indexTransaction(txn, - list(node1, node2, node3, node4), - list(nodeMetaData1, nodeMetaData2, nodeMetaData3, nodeMetaData4), - content); - - //Check for the TXN state stamp. - logger.info("#################### Started Second Test ##############################"); - builder = new BooleanQuery.Builder(); - builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_SOLR4_ID, "TRACKER!STATE!TX")), BooleanClause.Occur.MUST)); - builder.add(new BooleanClause(LegacyNumericRangeQuery.newLongRange(QueryConstants.FIELD_S_TXID, txn.getId(), txn.getId() + 1, true, false), BooleanClause.Occur.MUST)); - waitForQuery = builder.build(); - - waitForDocCount(waitForQuery, 1, MAX_WAIT_TIME); - logger.info("#################### Passed Second Test ##############################"); - - /* - * Query the index for the content - */ - - waitForDocCount(new TermQuery(new Term(QueryConstants.FIELD_READER, "jim")), 1, MAX_WAIT_TIME); - waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", token1)), 4, MAX_WAIT_TIME); - - logger.info("#################### Passed Third Test ##############################"); - - ModifiableSolrParams params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + node1.getId()); //Query for an id in the content field. The node id is automatically populated into the content field by test framework - params.add("qt", "/afts"); - params.add("fl", "DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - SolrServletRequest req = areq(params, null); - assertQ(req, "*[count(//doc)=4]", - "//result/doc[1]/long[@name='DBID'][.='" + node1.getId() + "']", - "//result/doc[2]/long[@name='DBID'][.='" + node3.getId() + "']", - "//result/doc[3]/long[@name='DBID'][.='" + node2.getId() + "']", - "//result/doc[4]/long[@name='DBID'][.='" + node4.getId() + "']"); - - params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + node1.getId() + "_70"); //Query for an id in the content field. The node id is automatically populated into the content field by test framework - params.add("qt", "/afts"); - params.add("fl","DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - req = areq(params, null); - assertQ(req, "*[count(//doc)= 2]", - "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", - "//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']"); - - params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + node1.getId()+"_45"); //Query for an id in the content field. The node id is automatically populated into the content field by test framework - params.add("qt", "/afts"); - params.add("fl","DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - req = areq(params, null); - assertQ(req, "*[count(//doc)= 3]", - "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", - "//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']", - "//result/doc[3]/long[@name='DBID'][.='"+node2.getId()+"']"); - - params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + node1.getId()+"_30"); - params.add("qt", "/afts"); - params.add("fl","DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - req = areq(params, null); - assertQ(req, "*[count(//doc)= 4]", - "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", - "//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']", - "//result/doc[3]/long[@name='DBID'][.='"+node2.getId()+"']", - "//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']"); - - - params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + node4.getId()); - params.add("qt", "/afts"); - params.add("fl","DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - req = areq(params, null); - assertQ(req, "*[count(//doc)= 4]", - "//result/doc[1]/long[@name='DBID'][.='"+node4.getId()+"']", - "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", - "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']", - "//result/doc[4]/long[@name='DBID'][.='"+node1.getId()+"']"); - - //Test nodeRef - - params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + nodeMetaData1.getNodeRef().getId()); //Query for an id in the content field. The node id is automatically populated into the content field by test framework - params.add("qt", "/afts"); - params.add("fl", "DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - - req = areq(params, null); - assertQ(req, "*[count(//doc)=4]", - "//result/doc[1]/long[@name='DBID'][.='" + node1.getId() + "']", - "//result/doc[2]/long[@name='DBID'][.='" + node3.getId() + "']", - "//result/doc[3]/long[@name='DBID'][.='" + node2.getId() + "']", - "//result/doc[4]/long[@name='DBID'][.='" + node4.getId() + "']"); - - params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + nodeMetaData1.getNodeRef().getId() + "_70"); //Query for an id in the content field. The node id is automatically populated into the content field by test framework - params.add("qt", "/afts"); - params.add("fl","DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - req = areq(params, null); - assertQ(req, "*[count(//doc)= 2]", - "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", - "//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']"); - - params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + nodeMetaData1.getNodeRef().getId()+"_45"); //Query for an id in the content field. The node id is automatically populated into the content field by test framework - params.add("qt", "/afts"); - params.add("fl","DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - req = areq(params, null); - assertQ(req, "*[count(//doc)= 3]", - "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", - "//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']", - "//result/doc[3]/long[@name='DBID'][.='"+node2.getId()+"']"); - - params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + nodeMetaData1.getNodeRef().getId()+"_30"); - params.add("qt", "/afts"); - params.add("fl","DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - req = areq(params, null); - assertQ(req, "*[count(//doc)= 4]", - "//result/doc[1]/long[@name='DBID'][.='"+node1.getId()+"']", - "//result/doc[2]/long[@name='DBID'][.='"+node3.getId()+"']", - "//result/doc[3]/long[@name='DBID'][.='"+node2.getId()+"']", - "//result/doc[4]/long[@name='DBID'][.='"+node4.getId()+"']"); - - - params = new ModifiableSolrParams(); - params.add("q", "FINGERPRINT:" + nodeMetaData4.getNodeRef().getId()); - params.add("qt", "/afts"); - params.add("fl","DBID,score"); - params.add("start", "0"); - params.add("rows", "6"); - req = areq(params, null); - assertQ(req, "*[count(//doc)= 4]", - "//result/doc[1]/long[@name='DBID'][.='"+node4.getId()+"']", - "//result/doc[2]/long[@name='DBID'][.='"+node2.getId()+"']", - "//result/doc[3]/long[@name='DBID'][.='"+node3.getId()+"']", - "//result/doc[4]/long[@name='DBID'][.='"+node1.getId()+"']"); - + return stream(sizes) + .mapToObj(item -> { + Random randomizer = new Random(1); + return range(0, item) + .mapToObj(i -> randomizer.nextInt()) + .map(Object::toString) + .collect(Collectors.joining(" "));}) + .collect(Collectors.toList()); } } diff --git a/search-services/alfresco-search/src/test/java/org/alfresco/solr/query/DistributedAlfrescoSolrFingerPrintTest.java b/search-services/alfresco-search/src/test/java/org/alfresco/solr/query/DistributedAlfrescoSolrFingerPrintTest.java index 12158f5eb..980ad45db 100644 --- a/search-services/alfresco-search/src/test/java/org/alfresco/solr/query/DistributedAlfrescoSolrFingerPrintTest.java +++ b/search-services/alfresco-search/src/test/java/org/alfresco/solr/query/DistributedAlfrescoSolrFingerPrintTest.java @@ -18,6 +18,7 @@ */ package org.alfresco.solr.query; +import static java.util.Arrays.asList; import static org.alfresco.solr.AlfrescoSolrUtils.getAcl; import static org.alfresco.solr.AlfrescoSolrUtils.getAclChangeSet; import static org.alfresco.solr.AlfrescoSolrUtils.getAclReaders; @@ -25,12 +26,13 @@ import static org.alfresco.solr.AlfrescoSolrUtils.getNode; import static org.alfresco.solr.AlfrescoSolrUtils.getNodeMetaData; import static org.alfresco.solr.AlfrescoSolrUtils.getTransaction; import static org.alfresco.solr.AlfrescoSolrUtils.indexAclChangeSet; -import static org.alfresco.solr.AlfrescoSolrUtils.list; +import static java.util.Collections.singletonList; import java.util.ArrayList; import java.util.List; import java.util.Random; +import org.alfresco.model.ContentModel; import org.alfresco.repo.search.adaptor.lucene.QueryConstants; import org.alfresco.solr.AbstractAlfrescoDistributedTest; import org.alfresco.solr.client.Acl; @@ -38,6 +40,7 @@ import org.alfresco.solr.client.AclChangeSet; import org.alfresco.solr.client.AclReaders; import org.alfresco.solr.client.Node; import org.alfresco.solr.client.NodeMetaData; +import org.alfresco.solr.client.StringPropertyValue; import org.alfresco.solr.client.Transaction; import org.apache.lucene.index.Term; import org.apache.lucene.search.BooleanClause; @@ -60,31 +63,26 @@ import org.junit.Test; @LuceneTestCase.SuppressCodecs({"Appending","Lucene3x","Lucene40","Lucene41","Lucene42","Lucene43", "Lucene44", "Lucene45","Lucene46","Lucene47","Lucene48","Lucene49"}) public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDistributedTest { - private static Node[] nodes = new Node[4]; - private static NodeMetaData[] nodesMetada = new NodeMetaData[4]; - + private static long MAX_WAIT_TIME = 80000; + + private static Node[] NODES = new Node[4]; + private static NodeMetaData[] NODES_METADATA = new NodeMetaData[4]; + private static Acl ACL; + @BeforeClass private static void initData() throws Throwable { - initSolrServers(2,"DistributedAlfrescoSolrFingerPrintTest",null); - /* - * Create and index an AclChangeSet. - */ + initSolrServers(2,getClassName(),null); AclChangeSet aclChangeSet = getAclChangeSet(1); - Acl acl = getAcl(aclChangeSet); + ACL = getAcl(aclChangeSet); Acl acl2 = getAcl(aclChangeSet); + AclReaders aclReaders = getAclReaders(aclChangeSet, ACL, singletonList("joel"), singletonList("phil"), null); + AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, singletonList("jim"), singletonList("phil"), null); - AclReaders aclReaders = getAclReaders(aclChangeSet, acl, list("joel"), list("phil"), null); - AclReaders aclReaders2 = getAclReaders(aclChangeSet, acl2, list("jim"), list("phil"), null); - - - indexAclChangeSet(aclChangeSet, - list(acl, acl2), - list(aclReaders, aclReaders2)); - + indexAclChangeSet(aclChangeSet, asList(ACL, acl2), asList(aclReaders, aclReaders2)); //Check for the ACL state stamp. BooleanQuery.Builder builder = new BooleanQuery.Builder(); @@ -93,27 +91,22 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist BooleanQuery waitForQuery = builder.build(); waitForDocCountAllCores(waitForQuery, 1, 80000); - /* - * Create and index a Transaction - */ - - //First create a transaction. Transaction txn = getTransaction(0, 4); - //Next create two nodes to update for the transaction - nodes[0] = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED); - nodes[1] = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED); - nodes[2] = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED); - nodes[3] = getNode(txn, acl, Node.SolrApiNodeStatus.UPDATED); + //Next create two NODES to update for the transaction + NODES[0] = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED); + NODES[1] = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED); + NODES[2] = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED); + NODES[3] = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED); //Next create the NodeMetaData for each node. TODO: Add more metadata - nodesMetada[0] = getNodeMetaData(nodes[0], txn, acl, "mike", null, false); - nodesMetada[1] = getNodeMetaData(nodes[1], txn, acl, "mike", null, false); - nodesMetada[2] = getNodeMetaData(nodes[2], txn, acl, "mike", null, false); - nodesMetada[3] = getNodeMetaData(nodes[3], txn, acl, "mike", null, false); + NODES_METADATA[0] = getNodeMetaData(NODES[0], txn, ACL, "mike", null, false); + NODES_METADATA[1] = getNodeMetaData(NODES[1], txn, ACL, "mike", null, false); + NODES_METADATA[2] = getNodeMetaData(NODES[2], txn, ACL, "mike", null, false); + NODES_METADATA[3] = getNodeMetaData(NODES[3], txn, ACL, "mike", null, false); - List content = new ArrayList(); + List content = new ArrayList<>(); int[] sizes = {2000, 1000, 1500, 750}; Random r = new Random(1); @@ -127,16 +120,16 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist if(s>0) { buf.append(" "); } - buf.append(Integer.toString(Math.abs(rand.nextInt()))); + buf.append(Math.abs(rand.nextInt())); } content.add(buf.toString()); } - //Index the transaction, nodes, and nodeMetaDatas. + //Index the transaction, NODES, and nodeMetaDatas. //Note that the content is automatically created by the test framework. indexTransaction(txn, - list(nodes[0], nodes[1], nodes[2], nodes[3]), - list(nodesMetada[0], nodesMetada[1], nodesMetada[2], nodesMetada[3]), + asList(NODES[0], NODES[1], NODES[2], NODES[3]), + asList(NODES_METADATA[0], NODES_METADATA[1], NODES_METADATA[2], NODES_METADATA[3]), content); //Check for the TXN state stamp. @@ -156,7 +149,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist } @AfterClass - private static void destroyData() throws Throwable + private static void destroyData() { dismissSolrServers(); } @@ -167,7 +160,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist putHandleDefaults(); QueryResponse response = query(getDefaultTestClient(), true, "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}", - params("q", "FINGERPRINT:"+nodes[0].getId(), + params("q", "FINGERPRINT:"+ NODES[0].getId(), "qt", "/afts", "shards.qt", "/afts", "start", "0", @@ -175,22 +168,22 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist "rows", "100")); SolrDocumentList docs = response.getResults(); - assertTrue(docs.getNumFound() == 4); + assertEquals(4, docs.getNumFound()); SolrDocument doc0 = docs.get(0); long dbid0 = (long)doc0.getFieldValue("DBID"); - assertTrue(dbid0 == nodes[0].getId()); + assertEquals(dbid0, NODES[0].getId()); SolrDocument doc1 = docs.get(1); long dbid1 = (long)doc1.getFieldValue("DBID"); - assertTrue(dbid1 == nodes[2].getId()); + assertEquals(dbid1, NODES[2].getId()); SolrDocument doc2 = docs.get(2); long dbid2 = (long)doc2.getFieldValue("DBID"); - assertTrue(dbid2 == nodes[1].getId()); + assertEquals(dbid2, NODES[1].getId()); SolrDocument doc3 = docs.get(3); long dbid3 = (long)doc3.getFieldValue("DBID"); - assertTrue(dbid3 == nodes[3].getId()); + assertEquals(dbid3, NODES[3].getId()); } @Test @@ -199,7 +192,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist putHandleDefaults(); QueryResponse response = query(getDefaultTestClient(), true, "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}", - params("q", "FINGERPRINT:" + nodes[0].getId()+"_70", + params("q", "FINGERPRINT:" + NODES[0].getId()+"_70", "qt", "/afts", "shards.qt", "/afts", "start", "0", @@ -207,14 +200,14 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist "rows", "100")); SolrDocumentList docs = response.getResults(); - assertTrue(docs.getNumFound() == 2); + assertEquals(2, docs.getNumFound()); SolrDocument doc0 = docs.get(0); long dbid0 = (long)doc0.getFieldValue("DBID"); - assertTrue(dbid0 == nodes[0].getId()); + assertEquals(dbid0, NODES[0].getId()); SolrDocument doc1 = docs.get(1); long dbid1 = (long)doc1.getFieldValue("DBID"); - assertTrue(dbid1 == nodes[2].getId()); + assertEquals(dbid1, NODES[2].getId()); } @Test @@ -223,7 +216,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist putHandleDefaults(); QueryResponse response = query(getDefaultTestClient(), true, "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}", - params("q", "FINGERPRINT:" + nodes[0].getId()+"_45", + params("q", "FINGERPRINT:" + NODES[0].getId()+"_45", "qt", "/afts", "shards.qt", "/afts", "start", "0", @@ -231,18 +224,18 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist "rows", "100")); SolrDocumentList docs = response.getResults(); - assertTrue(docs.getNumFound() == 3); + assertEquals(3, docs.getNumFound()); SolrDocument doc0 = docs.get(0); long dbid0 = (long)doc0.getFieldValue("DBID"); - assertTrue(dbid0 == nodes[0].getId()); + assertEquals(dbid0, NODES[0].getId()); SolrDocument doc1 = docs.get(1); long dbid1 = (long)doc1.getFieldValue("DBID"); - assertTrue(dbid1 == nodes[2].getId()); + assertEquals(dbid1, NODES[2].getId()); SolrDocument doc2 = docs.get(2); long dbid2 = (long)doc2.getFieldValue("DBID"); - assertTrue(dbid2 == nodes[1].getId()); + assertEquals(dbid2, NODES[1].getId()); } @Test @@ -251,7 +244,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist putHandleDefaults(); QueryResponse response = query(getDefaultTestClient(), true, "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}", - params("q", "FINGERPRINT:"+nodesMetada[0].getNodeRef().getId(), + params("q", "FINGERPRINT:"+ NODES_METADATA[0].getNodeRef().getId(), "qt", "/afts", "shards.qt", "/afts", "start", "0", @@ -259,22 +252,22 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist "rows", "100")); SolrDocumentList docs = response.getResults(); - assertTrue(docs.getNumFound() == 4); + assertEquals(4, docs.getNumFound()); SolrDocument doc0 = docs.get(0); long dbid0 = (long)doc0.getFieldValue("DBID"); - assertTrue(dbid0 == nodes[0].getId()); + assertEquals(dbid0, NODES[0].getId()); SolrDocument doc1 = docs.get(1); long dbid1 = (long)doc1.getFieldValue("DBID"); - assertTrue(dbid1 == nodes[2].getId()); + assertEquals(dbid1, NODES[2].getId()); SolrDocument doc2 = docs.get(2); long dbid2 = (long)doc2.getFieldValue("DBID"); - assertTrue(dbid2 == nodes[1].getId()); + assertEquals(dbid2, NODES[1].getId()); SolrDocument doc3 = docs.get(3); long dbid3 = (long)doc3.getFieldValue("DBID"); - assertTrue(dbid3 == nodes[3].getId()); + assertEquals(dbid3, NODES[3].getId()); } @Test @@ -283,7 +276,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist putHandleDefaults(); QueryResponse response = query(getDefaultTestClient(), true, "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}", - params("q", "FINGERPRINT:" + nodesMetada[0].getNodeRef().getId() +"_70", + params("q", "FINGERPRINT:" + NODES_METADATA[0].getNodeRef().getId() +"_70", "qt", "/afts", "shards.qt", "/afts", "start", "0", @@ -291,14 +284,14 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist "rows", "100")); SolrDocumentList docs = response.getResults(); - assertTrue(docs.getNumFound() == 2); + assertEquals(2, docs.getNumFound()); SolrDocument doc0 = docs.get(0); long dbid0 = (long)doc0.getFieldValue("DBID"); - assertTrue(dbid0 == nodes[0].getId()); + assertEquals(dbid0, NODES[0].getId()); SolrDocument doc1 = docs.get(1); long dbid1 = (long)doc1.getFieldValue("DBID"); - assertTrue(dbid1 == nodes[2].getId()); + assertEquals(dbid1, NODES[2].getId()); } @Test @@ -307,7 +300,7 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist putHandleDefaults(); QueryResponse response = query(getDefaultTestClient(), true, "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}", - params("q", "FINGERPRINT:" + nodesMetada[0].getNodeRef().getId() +"_45", + params("q", "FINGERPRINT:" + NODES_METADATA[0].getNodeRef().getId() +"_45", "qt", "/afts", "shards.qt", "/afts", "start", "0", @@ -315,20 +308,113 @@ public class DistributedAlfrescoSolrFingerPrintTest extends AbstractAlfrescoDist "rows", "100")); SolrDocumentList docs = response.getResults(); - assertTrue(docs.getNumFound() == 3); + assertEquals(3, docs.getNumFound()); SolrDocument doc0 = docs.get(0); long dbid0 = (long)doc0.getFieldValue("DBID"); - assertTrue(dbid0 == nodes[0].getId()); + assertEquals(dbid0, NODES[0].getId()); SolrDocument doc1 = docs.get(1); long dbid1 = (long)doc1.getFieldValue("DBID"); - assertTrue(dbid1 == nodes[2].getId()); + assertEquals(dbid1, NODES[2].getId()); SolrDocument doc2 = docs.get(2); long dbid2 = (long)doc2.getFieldValue("DBID"); - assertTrue(dbid2 == nodes[1].getId()); + assertEquals(dbid2, NODES[1].getId()); } - - -} + @Test + public void testFingerprintStillExistsAfterNodeMetadataUpdate() throws Exception + { + putHandleDefaults(); + + Transaction txn = getTransaction(0, 1); + Node fileNode = getNode(txn, ACL, Node.SolrApiNodeStatus.UPDATED); + NodeMetaData fileMetaData = getNodeMetaData(fileNode, txn, ACL, "mike", null, false); + + indexTransaction( + txn, + singletonList(fileNode), + singletonList(fileMetaData), + singletonList("This is a text content which is longer than the default hello world " + fileNode.getId() + + " returned by the Mock SOLRAPIQueueClient. This is needed because the \"min_hash\" field type " + + "definition in Solr doesn't take in account fields which produce less than 5 tokens (see the " + + "ShingleFilter settings).")); + + makeSureContentNodeHasBeenIndexed(fileNode, "mike", "longer"); + + QueryResponse response = query(getDefaultTestClient(), true, + "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}", + params("q", "FINGERPRINT:" + fileMetaData.getNodeRef().getId(), + "qt", "/afts", + "shards.qt", "/afts", + "start", "0", + "fl", "DBID,score", + "rows", "100")); + + SolrDocumentList docs = response.getResults(); + assertEquals(1, docs.getNumFound()); + assertEquals(fileNode.getId(), docs.iterator().next().getFieldValue("DBID")); + + // Let's update the test node + fileMetaData.setOwner("Andrea"); + fileMetaData.getProperties().put(ContentModel.PROP_TITLE, new StringPropertyValue("This is the new file \"title\" metadata attribute.")); + + txn = getTransaction(0, 1); + + indexTransaction( + txn, + singletonList(fileNode), + singletonList(fileMetaData)); + + makeSureContentNodeHasBeenIndexed(fileNode, "Andrea", "longer"); + + response = query(getDefaultTestClient(), true, + "{\"locales\":[\"en\"], \"templates\": [{\"name\":\"t1\", \"template\":\"%cm:content\"}], \"authorities\": [\"joel\"], \"tenants\": []}", + params("q", "FINGERPRINT:" + fileMetaData.getNodeRef().getId(), + "qt", "/afts", + "shards.qt", "/afts", + "start", "0", + "fl", "DBID,score", + "rows", "100")); + docs = response.getResults(); + assertEquals(1, docs.getNumFound()); + assertEquals(fileNode.getId(), docs.iterator().next().getFieldValue("DBID")); + } + + /** + * Queries the index using a token from the (dummy) text produced by the test framework ("world", actually). + * Once the query returns a positive result we are sure the ContentTracker + * + *
    + *
  1. + * Fetched the text content associated with the current node, from Alfresco + *
    2. + *
  2. + * Computed a fingerprint (using the retrieved text) for the node + *
    3. + *
  3. + * Updated the node definition in the (Solr)ContentStore and in Solr + *
    4. + *
+ * + * Last but not least, we are also making sure that CommitTracker executed its cycle as well (otherwise document + * wouldn't be searchable). + * + * @param node an addition term which will be appended as a required clause in the executed query. + * @param testTerm a term which is supposed to be in the indexed content + * @param owner the #FIELD_OWNER which will be used as an additional required query clause. + * @throws Exception in case the MAX_WAIT_TIME is reached and the node is not in results. + */ + private void makeSureContentNodeHasBeenIndexed(final Node node, final String owner, String testTerm) throws Exception + { + waitForDocCount(new TermQuery(new Term(QueryConstants.FIELD_READER, "jim")), 1, MAX_WAIT_TIME); + waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), 1, MAX_WAIT_TIME); + + waitForDocCount(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", Long.toString(node.getId()))), 1, MAX_WAIT_TIME); + + BooleanQuery.Builder builder = new BooleanQuery.Builder(); + builder.add(new BooleanClause(new TermQuery(new Term("content@s___t@{http://www.alfresco.org/model/content/1.0}content", testTerm)), BooleanClause.Occur.MUST)); + builder.add(new BooleanClause(new TermQuery(new Term(QueryConstants.FIELD_OWNER, owner)), BooleanClause.Occur.MUST)); + waitForDocCount(builder.build(), 1, MAX_WAIT_TIME); + } +} \ No newline at end of file diff --git a/search-services/alfresco-search/src/test/resources/test-files/collection1/conf/schema-fingerprint.xml b/search-services/alfresco-search/src/test/resources/test-files/collection1/conf/schema-fingerprint.xml deleted file mode 100644 index 8befa0480..000000000 --- a/search-services/alfresco-search/src/test/resources/test-files/collection1/conf/schema-fingerprint.xml +++ /dev/null @@ -1,769 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - id - -