diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..575155bab --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,67 @@ +# Contributing + +Thanks for your interest in contributing to this project! + +The following is a set of guidelines for contributing to this library. Most of them will +make the life of the reviewer easier and therefore decrease the time required for the +patch be included in the next version. + +Alfresco has an [active forum](http://community.alfresco.com/community/ecm) to support +community users of our products. If you have any questions then this is the fastest method +of getting an answer. + +## Raising issues + +If you want to raise an issue then please use the [issue tracker on GitHub](https://github.com/Alfresco/SearchServices/issues). +We may convert these to Jira tickets before working on them, as this is the system we use +internally for tracking development. The Jira project for this codebase is [SEARCH](https://issues.alfresco.com/jira/projects/SEARCH/issues) +and you may want to look here first to see if your issue has previously been encountered.[^jiraaccess] +There are some guidelines for raising a good issue [here](https://hub.alfresco.com/t5/alfresco-content-services-hub/reporting-an-issue/ba-p/289727). + +[^jiraaccess]: Note that while we try to keep our Jira issues visible to everyone, some +are restricted as they relate to specific customers or security issues. Some older issues +are also restricted simply because we have not been through to check if they contain +sensitive information or not. + +## Submitting changes + +We have a [coding standards guidelines page](https://hub.alfresco.com/t5/alfresco-content-services-hub/coding-standards-for-alfresco-content-services/ba-p/290457) +although you will find numerous examples where we have not adhered to them. Please try to +maintain consistency with the guidelines for new code, but avoid reformatting large +blocks of code if these are not related to your change. + +## Branches + +Our codebase consists of long-lived release branches and short-lived feature branches. The +code that we expect to include in the next minor version is stored on `master`. All other +release branches have the prefix `release/`. Feature branches may have any other prefix, +but we usually use `feature/` or `fix/`. We expect code on release branches to be ready +to release, and in the rare occasion when a release branch is broken then we try to revert +changes to fix the branch as soon as possible. + +As bug fixes often also need a change to ACS then we use a cherry-pick strategy to get the +fix to all necessary release branches. The fix should initially be merged to `master` and +it can then be cherry-picked back by using: + +```git cherry-pick -x -m 1 [mergeCommitId]``` + +## Community Mirror + +Pull requests to our community mirror will be accepted in our enterprise codebase and then +mirrored back to the community. You will always be credited with your commits, although if +you [sign your commits](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) then the +signature will be stripped by the mirroring process.[^dependabot] + +[^dependabot]: This is the reason that pull requests submitted by Dependabot appear closed +rather than merged. + +## Builds + +Our builds are currently in our internal Bamboo instance. If you have access to the +internal code then you can find some more links in the [insight engine module](insight-engine/CONTRIBUTING.md). + +Although the build results are not visible externally, it should be possible to run most of +the tests locally. We have divided our tests into unit tests, integration tests and +end-to-end tests. The unit and integration tests can be run using the maven `test` and +`verify` goals respectively. The end-to-end tests cannot currently be run externally as +they require some dependencies stored in our internal Nexus. diff --git a/README.md b/README.md index 0b78fd2e7..b973c9523 100644 --- a/README.md +++ b/README.md @@ -41,3 +41,7 @@ More details are available at [search-services](/search-services) folder. **Following resources will not be available for Community users** More details are available at [insight-engine](/insight-engine) folder. + +## Contributing guide + +Please use [this guide](CONTRIBUTING.md) to make a contribution to the project. diff --git a/e2e-test/generator-alfresco-docker-compose/README.md b/e2e-test/generator-alfresco-docker-compose/README.md index 986904543..0caf4d33d 100644 --- a/e2e-test/generator-alfresco-docker-compose/README.md +++ b/e2e-test/generator-alfresco-docker-compose/README.md @@ -111,7 +111,8 @@ When using Community, some different options can be combined: ? Would you like to use HTTP or mTLS for Alfresco-SOLR communication? http ? Would you like to use HTTP or HTTPs for Web Proxy? http ? Would you like to protect the access to SOLR REST API? Yes -? Would you like to use a SOLR Replication (2 nodes in master-slave)? No +? Would you like to use a SOLR Replication? No +? Would you like to compress Get Content responses? No ``` **Note** that when choosing *mTLS* or *HTTPs*, default certificates, truststores and keystores are provided for testing purposes. If you are planning to use this Docker Compose template for real environments, replace these cryptographic stores with another generated by yourself to increase the security of your system. diff --git a/e2e-test/generator-alfresco-docker-compose/generators/app/index.js b/e2e-test/generator-alfresco-docker-compose/generators/app/index.js index ac5b2f4a2..03905232a 100644 --- a/e2e-test/generator-alfresco-docker-compose/generators/app/index.js +++ b/e2e-test/generator-alfresco-docker-compose/generators/app/index.js @@ -56,7 +56,7 @@ module.exports = class extends Generator { type: 'confirm', name: 'protectSolr', message: 'Would you like to protect the access to SOLR REST API?', - default: 'true' + default: true }, { whenFunction: response => response.httpMode == 'http', @@ -69,6 +69,13 @@ module.exports = class extends Generator { { name: "Yes - two nodes in a master-master configuration", value: "master-master" } ] }, + { + whenFunction: response => response.acsVersion == '6.2', + type: 'confirm', + name: 'gzip', + message: 'Would you like to compress Get Content responses?', + default: false + }, // Enterprise only options { whenFunction: response => response.alfrescoVersion == 'enterprise' && !response.replication, @@ -208,7 +215,8 @@ module.exports = class extends Generator { searchPath: searchBasePath, zeppelin: (this.props.zeppelin ? "true" : "false"), sharding: (this.props.sharding ? "true" : "false"), - shardingMethod: (this.props.shardingMethod) + shardingMethod: (this.props.shardingMethod), + gzip: (this.props.gzip ? "true" : "false") } ); diff --git a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/.env b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/.env index b39fd3b1e..352f5b342 100755 --- a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/.env +++ b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/.env @@ -1,15 +1,15 @@ ALFRESCO_TAG=latest -SHARE_TAG=6.1.0-RC3 -POSTGRES_TAG=10.9 -TRANSFORM_ROUTER_TAG=1.1.0-EA2 -PDF_RENDERER_TAG=2.1.0-EA4 -IMAGE_MAGICK_TAG=2.1.0-EA4 -LIBREOFFICE_TAG=2.1.0-EA4 -TIKA_TAG=2.1.0-EA4 -TRANSFORM_MISC_TAG=2.1.0-EA4 +SHARE_TAG=6.2.0 +POSTGRES_TAG=11.4 +TRANSFORM_ROUTER_TAG=1.1.0 +PDF_RENDERER_TAG=2.1.0 +IMAGE_MAGICK_TAG=2.1.0 +LIBREOFFICE_TAG=2.1.0 +TIKA_TAG=2.1.0 +TRANSFORM_MISC_TAG=2.1.0 SHARED_FILE_STORE_TAG=0.5.3 ACTIVE_MQ_TAG=5.15.8 -DIGITAL_WORKSPACE_TAG=1.1.0 +DIGITAL_WORKSPACE_TAG=1.3.0 ACS_NGINX_TAG=3.0.1 ACS_COMMUNITY_NGINX_TAG=1.0.0 SEARCH_TAG=latest diff --git a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/docker-compose-ce.yml b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/docker-compose-ce.yml index 451e57945..7c137e4a0 100755 --- a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/docker-compose-ce.yml +++ b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/docker-compose-ce.yml @@ -13,6 +13,7 @@ services: TRUSTSTORE_PASS: kT9X6oe68t KEYSTORE_TYPE: JCEKS KEYSTORE_PASS: kT9X6oe68t <% } %> + COMPRESS_CONTENT: "<%=gzip%>" mem_limit: 1800m environment: JAVA_OPTS : " @@ -66,6 +67,7 @@ services: KEYSTORE_TYPE: JCEKS <% } %> <% if (replication) { %> ENABLE_MASTER: "true" ENABLE_SLAVE: "false" <% } %> + COMPRESS_CONTENT: "<%=gzip%>" mem_limit: 1200m environment: #Solr needs to know how to register itself with Alfresco @@ -108,6 +110,7 @@ services: ENABLE_MASTER: <% if (replication == 'master-master') { %>"true"<% } else { %>"false"<% } %> ENABLE_SLAVE: <% if (replication == 'master-master') { %>"false"<% } else { %>"true"<% } %> MASTER_HOST: solr6 <% } %> + COMPRESS_CONTENT: "<%=gzip%>" mem_limit: 1200m environment: #Solr needs to know how to register itself with Alfresco diff --git a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/images/alfresco/Dockerfile b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/images/alfresco/Dockerfile index a5595c7ac..87091db7a 100755 --- a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/images/alfresco/Dockerfile +++ b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/images/alfresco/Dockerfile @@ -40,13 +40,27 @@ RUN if [ "$SOLR_COMMS" == "https" ] ; then \ [[:space:]]\+<\/Engine>/\n\ <\/Engine>\n\ \n\ <\/Connector>/g" ${TOMCAT_DIR}/conf/server.xml; \ fi +# GZIP COMPRESSION +ARG COMPRESS_CONTENT +ENV COMPRESS_CONTENT $COMPRESS_CONTENT +RUN if [ "$COMPRESS_CONTENT" == "true" ] ; then \ + sed -i "s/\ +[[:space:]]\+connectionTimeout=\"20000\"/\n\ + connectionTimeout=\"20000\"\n\ + compression=\"on\"\n\ + compressionMinSize=\"1\"\n\ + /g" ${TOMCAT_DIR}/conf/server.xml; \ + fi + + # Copy custom content model to deployment folder COPY model/* $TOMCAT_DIR/shared/classes/alfresco/extension/ diff --git a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/images/search/Dockerfile b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/images/search/Dockerfile index 163f75682..0ff7861ba 100755 --- a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/images/search/Dockerfile +++ b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/images/search/Dockerfile @@ -125,6 +125,14 @@ RUN if [ "$ENABLE_SHARDING" == "true" ] ; then \ fi; \ fi +# GZIP COMPRESSION +ARG COMPRESS_CONTENT +ENV COMPRESS_CONTENT $COMPRESS_CONTENT +RUN if [ "$COMPRESS_CONTENT" == "true" ] ; then \ + sed -i '/^bash.*/i sed -i "'"s/solr.request.content.compress=false/solr.request.content.compress=true/g"'" ${DIST_DIR}/solrhome/templates/rerank/conf/solrcore.properties\n' \ + ${DIST_DIR}/solr/bin/search_config_setup.sh; \ +fi + # Useless for 'none'/'http' communications with Alfresco RUN mkdir ${DIST_DIR}/keystore \ && chown -R solr:solr ${DIST_DIR}/keystore diff --git a/e2e-test/pom.xml b/e2e-test/pom.xml index 5cee7b842..628c981b8 100644 --- a/e2e-test/pom.xml +++ b/e2e-test/pom.xml @@ -11,10 +11,11 @@ Search Analytics E2E Tests Test Project to test Search Service and Analytics Features on a complete setup of Alfresco, Share - 6.0.1.2 - 6.0.0.4 - 3.0.14 - 3.2.0 + 1.28 + 1.26 + 1.13 + 3.0.19 + 3.3.0 src/test/resources/SearchSuite.xml @@ -44,7 +45,7 @@ org.alfresco.tas - restapi-test + restapi ${tas.rest.api.version} test @@ -56,7 +57,7 @@ org.alfresco.tas - cmis-test + cmis ${tas.cmis.api.version} test @@ -71,6 +72,12 @@ alfresco-governance-services-automation-enterprise-rest-api ${rm.version} test + + + org.alfresco.tas + restapi-test + + com.fasterxml.jackson.core @@ -88,6 +95,10 @@ com.fasterxml.jackson.core jackson-databind + + org.alfresco.tas + restapi-test + diff --git a/e2e-test/src/main/java/org/alfresco/search/TestGroup.java b/e2e-test/src/main/java/org/alfresco/search/TestGroup.java index a149ab6f7..fbb90a64e 100644 --- a/e2e-test/src/main/java/org/alfresco/search/TestGroup.java +++ b/e2e-test/src/main/java/org/alfresco/search/TestGroup.java @@ -38,6 +38,7 @@ public class TestGroup public static final String ACS_61n = "ACS_61n"; // Alfresco Content Services 6.1 or above public static final String ACS_611n = "ACS_611n"; // Alfresco Content Services 6.1.1 or above public static final String ACS_62n = "ACS_62n"; // Alfresco Content Services 6.2 or above + public static final String ACS_63n = "ACS_63n"; // Alfresco Content Services 6.3 or above public static final String AGS_302 = "AGS_302"; // Alfresco governance Services 3.0.2 or above diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/AbstractE2EFunctionalTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/AbstractE2EFunctionalTest.java index 0f22c4acb..d65cfac7c 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/AbstractE2EFunctionalTest.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/AbstractE2EFunctionalTest.java @@ -6,6 +6,16 @@ */ package org.alfresco.test.search.functional; +import static java.util.Optional.ofNullable; + +import static lombok.AccessLevel.PROTECTED; +import static org.testng.Assert.assertEquals; + +import java.util.List; +import java.util.Set; +import java.util.stream.Collectors; + +import lombok.Getter; import org.alfresco.cmis.CmisWrapper; import org.alfresco.dataprep.ContentService; import org.alfresco.dataprep.SiteService.Visibility; @@ -38,19 +48,14 @@ import org.springframework.test.context.testng.AbstractTestNGSpringContextTests; import org.testng.annotations.BeforeClass; import org.testng.annotations.BeforeSuite; -import lombok.Getter; - -import static java.util.Optional.ofNullable; -import static lombok.AccessLevel.PROTECTED; - /** * @author meenal bhave */ -@ContextConfiguration("classpath:alfresco-search-e2e-context.xml") +@ContextConfiguration ("classpath:alfresco-search-e2e-context.xml") public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringContextTests { /** The number of retries that a query will be tried before giving up. */ - private static final int SEARCH_MAX_ATTEMPTS = 6; + protected static final int SEARCH_MAX_ATTEMPTS = 6; private static final Logger LOGGER = LogFactory.getLogger(); @@ -76,26 +81,26 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont protected CmisWrapper cmisApi; @Autowired - // @Getter(value = PROTECTED) + // @Getter(value = PROTECTED) protected DataUser dataUser; @Autowired - @Getter(value = PROTECTED) + @Getter (value = PROTECTED) private ContentService contentService; protected UserModel testUser, adminUserModel, testUser2; protected SiteModel testSite, testSite2; protected static String unique_searchString; - + protected static final String SEARCH_LANGUAGE_CMIS = "cmis"; - + protected enum SearchLanguage { CMIS, AFTS - } + } - @BeforeSuite(alwaysRun = true) + @BeforeSuite (alwaysRun = true) public void beforeSuite() throws Exception { super.springTestContextPrepareTestInstance(); @@ -105,7 +110,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont deployCustomModel("model/sharding-content-model.xml"); } - @BeforeClass(alwaysRun = true) + @BeforeClass (alwaysRun = true) public void setup() { serverHealth.assertServerIsOnline(); @@ -210,7 +215,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont customModel.setNodeRef(modelInRepo.getId()); customModel.setNodeRef(customModel.getNodeRefWithoutVersion()); customModel.setCmisLocation(String.format("/Data Dictionary/Models/%s", fileName)); - LOGGER.info("Custom Model file: " + customModel.getCmisLocation()); + LOGGER.info("Custom Model file: " + customModel.getCmisLocation()); } else { @@ -240,9 +245,9 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Wait for Solr to finish indexing and search to return appropriate results - * - * @param userQuery Search Query - * @param contentToFind that's expected to be included / excluded from the results + * + * @param userQuery Search Query + * @param contentToFind that's expected to be included / excluded from the results * @param expectedInResults Whether we expect the content in the results or not. * @return true if search returns expected results, i.e. is given content is found or excluded from the results */ @@ -278,26 +283,25 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont return false; } - - /** - * Method to check if the contentName is returned in the SearchResponse - * @param response - * @param contentName - * @return - */ - public boolean isContentInSearchResponse(SearchResponse response, String contentName) - { - boolean found = response.getEntries().stream() - .map(entry -> entry.getModel().getName()) - .filter(name -> name.equalsIgnoreCase(contentName) || contentName.isBlank()).count() > 0; - return found; + /** + * Method to check if the contentName is returned in the SearchResponse. + * + * @param response the search response + * @param contentName the text we are using as matching/verifying criteria. + * @return true if if the item with the contentName text is returned in the SearchResponse. + */ + public boolean isContentInSearchResponse(SearchResponse response, String contentName) + { + return response.getEntries().stream() + .map(entry -> entry.getModel().getName()) + .anyMatch(name -> name.equalsIgnoreCase(contentName) || contentName.isBlank()); } /** * Wait for Solr to finish indexing: Indexing has caught up = true if search returns appropriate results - * - * @param userQuery: search query, this can include the fieldname, unique search string will guarantee accurate results + * + * @param userQuery: search query, this can include the fieldname, unique search string will guarantee accurate results * @param expectedInResults, true if entry is expected in the results set * @return true (indexing is finished) if search returns appropriate results */ @@ -309,7 +313,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * waitForIndexing method that matches / waits for filename, metadata to be indexed. - * + * * @param userQuery * @param expectedInResults * @return @@ -322,7 +326,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * waitForIndexing method that matches / waits for content to be indexed, this can take longer than metadata indexing. * Since Metadata is indexed first, use this method where tests, queries need content to be indexed too. - * + * * @param userQuery * @param expectedInResults * @return @@ -334,9 +338,9 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Wait for Solr to finish indexing: Indexing has caught up = true if search returns appropriate results - * - * @param fieldName: specific field to search for, e.g. name. When specified, the query will become: name:'userQuery' - * @param userQuery: search string, unique search string will guarantee accurate results + * + * @param fieldName: specific field to search for, e.g. name. When specified, the query will become: name:'userQuery' + * @param userQuery: search string, unique search string will guarantee accurate results * @param expectedInResults, true if entry is expected in the results set * @return true (indexing is finished) if search returns appropriate results */ @@ -349,7 +353,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Run a search as admin user and return the response - * + * * @param queryString: string to search for, unique search string will guarantee accurate results * @return the search response from the API */ @@ -360,8 +364,8 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Run a search as given user and return the response - * - * @param user: UserModel for the user you wish to run the query as + * + * @param user: UserModel for the user you wish to run the query as * @param queryString: string to search for, unique search string will guarantee accurate results * @return the search response from the API */ @@ -373,11 +377,11 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont searchRequest.setQuery(queryModel); return restClient.authenticateUser(user).withSearchAPI().search(searchRequest); } - + /** * Run a search with Spellcheck as given user and return the response - * @param user UserModel for the user you wish to run the query as - * @param queryModel The queryModel to search for, containing the query + * @param user UserModel for the user you wish to run the query as + * @param queryModel The queryModel to search for, containing the query * @param spellcheckQuery The Spellcheck Model containing the query * @return the search response from the API */ @@ -418,32 +422,22 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont query.setQuery(queryReq); return query; } - + protected DataUser getDataUser() { return dataUser; } - + /** * Helper method to test if the search query works and count matches where provided - * @param query: AFTS or cmis query string - * @param expectedCount: Only successful response is checked, when expectedCount is null (can not be exactly specified), - * @param setCmis: Query language is set to cmis when setCmis is true, AFTS when false + * + * @param query: AFTS or cmis query string + * @param expectedCount: Only successful response is checked, when expectedCount is null (can not be exactly specified), * @return SearchResponse */ protected SearchResponse testSearchQuery(String query, Integer expectedCount, SearchLanguage queryLanguage) { - RestRequestQueryModel queryModel = new RestRequestQueryModel(); - queryModel.setQuery(query); - - if (ofNullable(queryLanguage).isPresent()) - { - queryModel.setLanguage(queryLanguage.toString()); - } - - SearchResponse response = queryAsUser(testUser, queryModel); - - restClient.assertStatusCodeIs(HttpStatus.OK); + SearchResponse response = performSearch(testUser, query, queryLanguage); if (ofNullable(expectedCount).isPresent()) { @@ -452,7 +446,64 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont return response; } + + /** + * Helper method to test if the search query returns the expected results in the order given. + * + * @param query: AFTS or cmis query string + * @param expectedNames: The ordered list of names expected to be returned, + * @return SearchResponse + */ + protected SearchResponse testSearchQueryOrdered(String query, List expectedNames, SearchLanguage queryLanguage) + { + SearchResponse response = performSearch(testUser, query, queryLanguage); + + List names = response.getEntries().stream().map(s -> s.getModel().getName()).collect(Collectors.toList()); + + // Include lists in failure message as TestNG won't do this for lists. + assertEquals(names, expectedNames, "Unexpected results for query: " + query + " Expected: " + expectedNames + " but got " + names); + return response; + } + + /** + * Helper method to test if the search query returns the expected set of results. + * + * @param query: AFTS or cmis query string + * @param expectedNames: The ordered list of names expected to be returned, + * @return SearchResponse + */ + protected SearchResponse testSearchQueryUnordered(String query, Set expectedNames, SearchLanguage queryLanguage) + { + SearchResponse response = performSearch(testUser, query, queryLanguage); + + Set names = response.getEntries().stream().map(s -> s.getModel().getName()).collect(Collectors.toSet()); + + assertEquals(names, expectedNames, "Unexpected results for query: " + query); + + return response; + } + + private SearchResponse performSearch(UserModel asUser, String query, SearchLanguage queryLanguage) + { + RestRequestQueryModel queryModel = new RestRequestQueryModel(); + queryModel.setQuery(query); + + if (!ofNullable(asUser).isPresent()) + { + asUser = testUser; + } + + if (ofNullable(queryLanguage).isPresent()) + { + queryModel.setLanguage(queryLanguage.toString()); + } + + SearchResponse response = queryAsUser(asUser, queryModel); + + return response; + } + /** * Method to create and run a simple spellcheck query * When a spellcheck query is run a user, the query inputed and the user query is inputted @@ -464,8 +515,8 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont { RestRequestSpellcheckModel spellCheck = new RestRequestSpellcheckModel(); spellCheck.setQuery(userQuery); - - UserModel searchUser = ofNullable(user).isPresent()? user: testUser; + + UserModel searchUser = ofNullable(user).isPresent() ? user : testUser; SearchRequest searchReq = new SearchRequest(); RestRequestQueryModel queryReq = new RestRequestQueryModel(); queryReq.setQuery(query); @@ -478,8 +529,8 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Method to check the spellcheck object returned in the Search Response - * @param response SearchResponse - * @param spellCheckType String Values: searchInsteadFor, didYouMean or null + * @param response SearchResponse + * @param spellCheckType String Values: searchInsteadFor, didYouMean or null * @param spellCheckSuggestion String Values: suggestion string or null */ public void testSearchSpellcheckResponse(SearchResponse response, String spellCheckType, String spellCheckSuggestion) @@ -498,6 +549,6 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont { response.getContext().assertThat().field("spellCheck").isNotEmpty(); response.getContext().getSpellCheck().assertThat().field("suggestions").contains(spellCheckSuggestion); - } + } } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/AbstractCmisE2ETest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/AbstractCmisE2ETest.java index 3dc7ed8c3..f096c15ff 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/AbstractCmisE2ETest.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/AbstractCmisE2ETest.java @@ -61,7 +61,7 @@ public abstract class AbstractCmisE2ETest extends AbstractE2EFunctionalTest protected boolean waitForIndexing(String query, long expectedCountResults) { - for (int searchCount = 1; searchCount <= 3; searchCount++) + for (int searchCount = 1; searchCount <= SEARCH_MAX_ATTEMPTS; searchCount++) { try @@ -71,7 +71,7 @@ public abstract class AbstractCmisE2ETest extends AbstractE2EFunctionalTest } catch (AssertionError ae) { - LOGGER.debug(ae.toString()); + LOGGER.info(String.format("WaitForIndexing in Progress: %s", ae.toString())); } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByAspectTests.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByAspectTests.java index 8e0e0eb08..5228c38e6 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByAspectTests.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByAspectTests.java @@ -7,6 +7,7 @@ import org.alfresco.utility.data.provider.XMLTestDataProvider; import org.alfresco.utility.model.FileModel; import org.alfresco.utility.model.FolderModel; import org.alfresco.utility.model.QueryModel; +import org.testng.Assert; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; @@ -135,6 +136,7 @@ public class SolrSearchByAspectTests extends AbstractCmisE2ETest .replace("NODE_REF[f1]", tasFolder1.getNodeRef()) .replace("NODE_REF[s1]", siteDoclibNodeRef); - cmisApi.authenticateUser(testUser).withQuery(currentQuery).assertResultsCount().equals(query.getResults()); + cmisApi.authenticateUser(testUser); + Assert.assertTrue(waitForIndexing(currentQuery, query.getResults()), String.format("Result count not as expected for query: %s", currentQuery)); } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByIdTests.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByIdTests.java index 5d1026e29..8d10955fd 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByIdTests.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByIdTests.java @@ -7,6 +7,7 @@ import org.alfresco.utility.data.provider.XMLTestDataProvider; import org.alfresco.utility.model.FileModel; import org.alfresco.utility.model.FolderModel; import org.alfresco.utility.model.QueryModel; +import org.testng.Assert; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; @@ -87,7 +88,7 @@ public class SolrSearchByIdTests extends AbstractCmisE2ETest @Test(dataProviderClass = XMLTestDataProvider.class, dataProvider = "getQueriesData") @XMLDataConfig(file = "src/test/resources/testdata/search-by-id.xml") - public void executeSearchByAspect(QueryModel query) throws Exception + public void executeSearchById(QueryModel query) throws Exception { String currentQuery = query.getValue() .replace("NODE_REF[siteId]", siteDoclibNodeRef) @@ -96,6 +97,7 @@ public class SolrSearchByIdTests extends AbstractCmisE2ETest .replace("NODE_REF[f1]", tasFolder1.getNodeRef()) .replace("NODE_REF[f1-1]", tasSubFolder1.getNodeRef()); - cmisApi.authenticateUser(testUser).withQuery(currentQuery).assertResultsCount().equals(query.getResults()); + cmisApi.authenticateUser(testUser); + Assert.assertTrue(waitForIndexing(currentQuery, query.getResults()), String.format("Result count not as expected for query: %s", currentQuery)); } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPathTests.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPathTests.java index c1e0ae1b9..1c86b2912 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPathTests.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPathTests.java @@ -7,6 +7,7 @@ import org.alfresco.utility.data.provider.XMLTestDataProvider; import org.alfresco.utility.model.QueryModel; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.testng.Assert; import org.testng.annotations.AfterClass; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; @@ -51,6 +52,8 @@ public class SolrSearchByPathTests extends AbstractCmisE2ETest @XMLDataConfig(file = "src/test/resources/testdata/search-by-path.xml") public void executeSearchByPathQueries(QueryModel query) { - cmisApi.withQuery(query.getValue()).assertResultsCount().equals(query.getResults()); + cmisApi.authenticateUser(testUser); + Assert.assertTrue(waitForIndexing(query.getValue(), query.getResults()), String.format("Result count not as expected for query: %s", query.getValue())); + } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPropertyTests.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPropertyTests.java index 40ea49483..9b6ad81e7 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPropertyTests.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPropertyTests.java @@ -1,5 +1,8 @@ package org.alfresco.test.search.functional.searchServices.cmis; +import java.util.List; +import java.util.Set; + import org.alfresco.utility.Utility; import org.alfresco.utility.data.CustomObjectTypeProperties; import org.alfresco.utility.data.provider.XMLDataConfig; @@ -7,6 +10,7 @@ import org.alfresco.utility.data.provider.XMLTestDataProvider; import org.alfresco.utility.model.FileModel; import org.alfresco.utility.model.FolderModel; import org.alfresco.utility.model.QueryModel; +import org.testng.Assert; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; @@ -15,7 +19,7 @@ public class SolrSearchByPropertyTests extends AbstractCmisE2ETest private FolderModel guestf, tesf, restf, testtttf, testf, testf1, testf2, testf3, testf4; private FileModel guestc, restc, tesc, testtttc, testc, testc1, testc2, testc3; - @BeforeClass(alwaysRun = true) + @BeforeClass (alwaysRun = true) public void dataPreparation() throws Exception { dataContent.usingAdmin().deployContentModel("model/tas-model.xml"); @@ -46,68 +50,465 @@ public class SolrSearchByPropertyTests extends AbstractCmisE2ETest // Sites dataContent.usingUser(testUser).usingSite(testSite).createCustomContent(guestf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "guestf text") - .addProperty("tas:IntPropertyF", 222)); + .addProperty("tas:IntPropertyF", 222)); dataContent.usingUser(testUser).usingSite(testSite).createCustomContent(tesf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "tesf text") - .addProperty("tas:IntPropertyF", 224)); + .addProperty("tas:IntPropertyF", 224)); // Sites >> Folders dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(restf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "restf text") - .addProperty("tas:IntPropertyF", 223)); + .addProperty("tas:IntPropertyF", 223)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testtttf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testtttf text") - .addProperty("tas:IntPropertyF", 225)); + .addProperty("tas:IntPropertyF", 225)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf text") - .addProperty("tas:IntPropertyF", 226)); + .addProperty("tas:IntPropertyF", 226)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf1, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf1 text") - .addProperty("tas:IntPropertyF", 2221)); + .addProperty("tas:IntPropertyF", 2221)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf2, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf2 text") - .addProperty("tas:IntPropertyF", 2222)); + .addProperty("tas:IntPropertyF", 2222)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf3, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf3 text") - .addProperty("tas:IntPropertyF", 2223)); + .addProperty("tas:IntPropertyF", 2223)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf4, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf4 text") - .addProperty("tas:IntPropertyF", 2224)); + .addProperty("tas:IntPropertyF", 2224)); // Sites >> Files dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(guestc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "guestc text") - .addProperty("tas:IntPropertyC", 222)); + .addProperty("tas:IntPropertyC", 222)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(restc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "restc text") - .addProperty("tas:IntPropertyC", 223)); + .addProperty("tas:IntPropertyC", 223)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(tesc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "tesc text") - .addProperty("tas:IntPropertyC", 224)); + .addProperty("tas:IntPropertyC", 224)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testtttc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testtttc text") - .addProperty("tas:IntPropertyC", 225)); + .addProperty("tas:IntPropertyC", 225)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testc text") - .addProperty("tas:IntPropertyC", 226)); + .addProperty("tas:IntPropertyC", 226)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testc1, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testc1 text") - .addProperty("tas:IntPropertyC", 2221)); + .addProperty("tas:IntPropertyC", 2221)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testc2, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testc2 text") - .addProperty("tas:IntPropertyC", 2222)); + .addProperty("tas:IntPropertyC", 2222)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testc3, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testc3 text") - .addProperty("tas:IntPropertyC", 2223)); + .addProperty("tas:IntPropertyC", 2223)); // wait for solr index - Utility.waitToLoopTime(getSolrWaitTimeInSeconds()); + cmisApi.authenticateUser(testUser); + waitForIndexing("SELECT * FROM tas:document where cmis:name = 'testc3.txt'", 1); } - @Test(dataProviderClass = XMLTestDataProvider.class, dataProvider = "getQueriesData") - @XMLDataConfig(file = "src/test/resources/testdata/search-by-property.xml") - public void executeSearchByProperty(QueryModel query) throws Exception + @Test + public void testFileNameEquality() { - cmisApi.authenticateUser(testUser).withQuery(query.getValue()).assertResultsCount().equals(query.getResults()); + String query = "SELECT * FROM tas:document where cmis:name = 'testc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt")); + } + + @Test + public void testFileNameInequality() + { + String query = "SELECT * FROM tas:document where cmis:name <> 'testc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "restc.txt", "tesc.txt", "testtttc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileNameIn() + { + String query = "SELECT * FROM tas:document where cmis:name IN('testc.txt', 'guestc.txt', 'restc.txt')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt", "guestc.txt", "restc.txt")); + } + + @Test + public void testFileNameNotIn() + { + // Nb. "gustc" is missing an "e". + String query = "SELECT * FROM tas:document where cmis:name NOT IN('testc.txt', 'gustc.txt', 'restc.txt')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "tesc.txt", "testtttc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileNameLike() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE '%testc%'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileNameLikeExact() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE 'testc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt")); + } + + @Test + public void testFileNamePrefixSuffix() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE 't%tc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt", "testtttc.txt")); + } + + @Test + public void testFileNameUnderscore() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE 't__tc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt")); + } + + @Test + public void testFolderNameEquality() + { + String query = "SELECT * FROM tas:folder where cmis:name = 'testf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf")); + } + + @Test + public void testFolderNameInequality() + { + String query = "SELECT * FROM tas:folder where cmis:name <> 'testf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "restf", "testtttf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderNameIn() + { + String query = "SELECT * FROM tas:folder where cmis:name IN('testf', 'guestf', 'restf')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf", "guestf", "restf")); + } + + @Test + public void testFolderNameNotIn() + { + // Nb. "gustc" is missing an "e". + String query = "SELECT * FROM tas:folder where cmis:name NOT IN('testf', 'gustf', 'restf')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "testtttf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderNameLike() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE '%testf%'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderNameLikeExact() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE 'testf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf")); + } + + @Test + public void testFolderNamePrefixSuffix() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE 't%tf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttf", "testf")); + } + + @Test + public void testFolderNameUnderscore() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE 't__tf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf")); + } + + @Test + public void testFileNameOrderAsc() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE '%testc%' ORDER BY cmis:name ASC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testc1.txt", "testc2.txt", "testc3.txt", "testc.txt")); + } + + @Test + public void testFileNameOrderDesc() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE '%testc%' ORDER BY cmis:name DESC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testc.txt", "testc3.txt", "testc2.txt", "testc1.txt")); + } + + @Test + public void testFileOrderNameDescDateAsc() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE '%testc%' ORDER BY cmis:name DESC, cmis:creationDate ASC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testc.txt", "testc3.txt", "testc2.txt", "testc1.txt")); + } + + @Test + public void testFolderNameOrderAsc() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE '%testf%' ORDER BY cmis:name ASC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderNameOrderDesc() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE '%testf%' ORDER BY cmis:name DESC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testf4", "testf3", "testf2", "testf1", "testf")); + } + + @Test + public void testFolderOrderNameDescDateAsc() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE '%testf%' ORDER BY cmis:name DESC, cmis:creationDate ASC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testf4", "testf3", "testf2", "testf1", "testf")); + } + + @Test + public void testFileCustomPropertyEquality() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC = 'restc text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restc.txt")); + } + + @Test + public void testFileCustomPropertyInequality() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC <> 'testc1 text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "restc.txt", "tesc.txt", "testtttc.txt", "testc.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileCustomPropertyIn() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC IN('restc text', 'testc2 text', 'testc3 text')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restc.txt", "testc2.txt", "testc3.txt")); + } + + + @Test + public void testFileCustomPropertyNotIn() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC NOT IN('restc text', 'testc2 text', 'testc3 text')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "tesc.txt", "testtttc.txt", "testc.txt", "testc1.txt")); + } + + @Test + public void testFileCustomPropertyLike() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC LIKE '%restc%'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restc.txt")); + } + + @Test + public void testFileCustomPropertyUnderscore() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC LIKE 't__tc text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt")); + } + + @Test + public void testFolderCustomPropertyEquality() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF = 'restf text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restf")); + } + + @Test + public void testFolderCustomPropertyInequality() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF <> 'testf1 text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "restf", "testtttf", "testf", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderCustomPropertyIn() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF IN('restf text', 'testf2 text', 'testf3 text')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restf", "testf2", "testf3")); + } + + @Test + public void testFolderCustomPropertyNotIn() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF NOT IN('restf text', 'testf2 text', 'testf3 text')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "testtttf", "testf", "testf1", "testf4")); + } + + @Test + public void testFolderCustomPropertyLike() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF LIKE '%restf%'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restf")); + } + + @Test + public void testFolderCustomPropertyUnderscore() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF LIKE 't__tf text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf")); + } + + @Test + public void testFileIntEquality() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC = '222'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt")); + } + + @Test + public void testFileIntInequality() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC <> '223'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "tesc.txt", "testtttc.txt", "testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileIntLessThen() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC < '223'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt")); + } + + @Test + public void testFileIntLessThanOrEqual() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC <= '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "restc.txt", "tesc.txt")); + } + + @Test + public void testFileIntGreaterThanOrEqual() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC >= '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("tesc.txt", "testtttc.txt", "testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileIntGreaterThan() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC > '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttc.txt", "testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileIntIn() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC IN('222', '223', '224', '225')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "restc.txt", "tesc.txt", "testtttc.txt")); + } + + @Test + public void testFileIntNotIn() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC NOT IN('222', '223', '224')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttc.txt", "testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFolderIntEquality() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF = '222'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf")); + } + + @Test + public void testFolderIntInequality() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF <> '223'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "testtttf", "testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderIntLessThan() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF < '223'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf")); + } + + @Test + public void testFolderIntLessThanOrEqual() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF <= '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "restf")); + } + + @Test + public void testFolderIntGreaterThanOrEqual() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF >= '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("tesf", "testtttf", "testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderIntGreaterThan() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF > '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttf", "testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderIntIn() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF IN('222', '223', '224', '225')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "restf", "testtttf")); + } + + @Test + public void testFolderIntNotIn() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF NOT IN('222', '223', '224')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttf", "testf", "testf1", "testf2", "testf3", "testf4")); } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchInFolderTests.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchInFolderTests.java index 17647a6ed..bc903f30d 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchInFolderTests.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchInFolderTests.java @@ -7,6 +7,7 @@ import org.alfresco.utility.model.FileModel; import org.alfresco.utility.model.FileType; import org.alfresco.utility.model.FolderModel; import org.alfresco.utility.model.QueryModel; +import org.testng.Assert; import org.testng.annotations.AfterClass; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; @@ -55,6 +56,7 @@ public class SolrSearchInFolderTests extends AbstractCmisE2ETest public void executeCMISQuery(QueryModel query) throws Exception { String currentQuery = String.format(query.getValue(), parentFolder.getNodeRef()); - cmisApi.withQuery(currentQuery).assertResultsCount().equals(query.getResults()); + cmisApi.authenticateUser(testUser); + Assert.assertTrue(waitForIndexing(currentQuery, query.getResults()), String.format("Result count not as expected for query: %s", currentQuery)); } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchInTreeTests.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchInTreeTests.java index 3fd68cfa4..cf6647e86 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchInTreeTests.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchInTreeTests.java @@ -7,6 +7,7 @@ import org.alfresco.utility.model.FileModel; import org.alfresco.utility.model.FileType; import org.alfresco.utility.model.FolderModel; import org.alfresco.utility.model.QueryModel; +import org.testng.Assert; import org.testng.annotations.AfterClass; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; @@ -59,7 +60,7 @@ public class SolrSearchInTreeTests extends AbstractCmisE2ETest public void executeCMISQuery(QueryModel query) throws Exception { String currentQuery = String.format(query.getValue(), parentFolder.getNodeRef()); - cmisApi.withQuery(currentQuery) - .assertResultsCount().equals(query.getResults()); + cmisApi.authenticateUser(testUser); + Assert.assertTrue(waitForIndexing(currentQuery, query.getResults()), String.format("Result count not as expected for query: %s", currentQuery)); } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchScoreQueryTests.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchScoreQueryTests.java index 85b597fea..847c5bc1b 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchScoreQueryTests.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchScoreQueryTests.java @@ -9,6 +9,7 @@ import org.alfresco.utility.data.provider.XMLTestDataProvider; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.testng.Assert; import org.testng.annotations.AfterClass; import org.testng.annotations.Test; @@ -54,8 +55,8 @@ public class SolrSearchScoreQueryTests extends AbstractCmisE2ETest this.testData = testData; this.testData.createUsers(dataUser); this.testData.createSitesStructure(dataSite, dataContent, dataUser); - cmisApi.authenticateUser(dataUser.getCurrentUser()); - + testUser = dataUser.getCurrentUser(); + cmisApi.authenticateUser(testUser); } /** @@ -70,68 +71,50 @@ public class SolrSearchScoreQueryTests extends AbstractCmisE2ETest + "FROM cmis:document " + "WHERE CONTAINS('Quidditch') " + "ORDER BY orderCriteria"; - - if (waitForIndexing(query, 3)) - { - cmisApi - .withQuery(query) - .assertColumnIsOrdered().isOrderedAsc("orderCriteria"); - } - else - { - throw new AssertionError("Wait for indexing has failed!"); - } - + + Assert.assertTrue(waitForIndexing(query, 3), String.format("Result count not as expected for query: %s", query)); + + cmisApi.withQuery(query).assertColumnIsOrdered().isOrderedAsc("orderCriteria"); + } - - /** - * Verify that results are inverse ordered - * @throws Exception - */ - @Test(dependsOnMethods = "prepareDataForScoreSearch") + + /** + * Verify that results are inverse ordered + * + * @throws Exception + */ + @Test(dependsOnMethods = "prepareDataForScoreSearch") public void scoreQueryOrderedDesc() throws Exception { - String query = "SELECT cmis:objectId, SCORE() AS orderCriteria " + String query = "SELECT cmis:objectId, SCORE() AS orderCriteria " + "FROM cmis:document " + "WHERE CONTAINS('Quidditch') " + "ORDER BY orderCriteria DESC"; - - if (waitForIndexing(query, 3)) - { - cmisApi - .withQuery(query).assertColumnIsOrdered().isOrderedDesc("orderCriteria"); - } - else - { - throw new AssertionError("Wait for indexing has failed!"); - } - + + Assert.assertTrue(waitForIndexing(query, 3), String.format("Result count not as expected for query: %s", query)); + + cmisApi.withQuery(query).assertColumnIsOrdered().isOrderedDesc("orderCriteria"); + } - - /** - * Verify that all SCORE results are between 0 and 1 - * @throws Exception - */ - @Test(groups = { TestGroup.ACS_62n }, dependsOnMethods = "prepareDataForScoreSearch") + + /** + * Verify that all SCORE results are between 0 and 1 + * + * @throws Exception + */ + @Test(groups = { TestGroup.ACS_62n }, dependsOnMethods = "prepareDataForScoreSearch") public void scoreQueryInRange() throws Exception { - - String query = "SELECT cmis:objectId, SCORE() " - + "FROM cmis:document " - + "WHERE CONTAINS('Quidditch')"; - - if (waitForIndexing(query, 3)) - { - cmisApi - .withQuery(query) - .assertColumnValuesRange().isReturningValuesInRange("SEARCH_SCORE", BigDecimal.ZERO, BigDecimal.ONE); - } - else - { - throw new AssertionError("Wait for indexing has failed!"); - } + + String query = "SELECT cmis:objectId, SCORE() " + + "FROM cmis:document " + + "WHERE CONTAINS('Quidditch')"; + Assert.assertTrue(waitForIndexing(query, 3), String.format("Result count not as expected for query: %s", query)); + + cmisApi.withQuery(query).assertColumnValuesRange().isReturningValuesInRange("SEARCH_SCORE", BigDecimal.ZERO, BigDecimal.ONE); + } /** @@ -143,45 +126,33 @@ public class SolrSearchScoreQueryTests extends AbstractCmisE2ETest { String query = "SELECT cmis:objectId, SCORE() AS orderCriteria " - + "FROM cmis:document " - + "WHERE CONTAINS('Quidditch')"; - - if (waitForIndexing(query, 3)) - { - cmisApi - .withQuery(query) - .assertColumnValuesRange().isReturningValuesInRange("orderCriteria", BigDecimal.ZERO, BigDecimal.ONE); - } - else - { - throw new AssertionError("Wait for indexing has failed!"); - } - + + "FROM cmis:document " + + "WHERE CONTAINS('Quidditch')"; + + Assert.assertTrue(waitForIndexing(query, 3), String.format("Result count not as expected for query: %s", query)); + + cmisApi.withQuery(query).assertColumnValuesRange().isReturningValuesInRange("orderCriteria", BigDecimal.ZERO, BigDecimal.ONE); + } - /** - * Verify that SCORE is valid name for an alias - * Currently only supported with double quotes - * @throws Exception - */ - @Test(dependsOnMethods = "prepareDataForScoreSearch") + /** + * Verify that SCORE is valid name for an alias + * Currently only supported with double quotes + * + * @throws Exception + */ + @Test(dependsOnMethods = "prepareDataForScoreSearch") public void scoreQueryScoreAsAlias() throws Exception { - - String query = "SELECT cmis:objectId, SCORE() AS \"score\" " - + "FROM cmis:document " - + "WHERE CONTAINS('Quidditch')"; - - if (waitForIndexing(query, 3)) - { - cmisApi - .withQuery(query).assertResultsCount().equals(3); - } - else - { - throw new AssertionError("Wait for indexing has failed!"); - } - + + String query = "SELECT cmis:objectId, SCORE() AS \"score\" " + + "FROM cmis:document " + + "WHERE CONTAINS('Quidditch')"; + + Assert.assertTrue(waitForIndexing(query, 3), String.format("Result count not as expected for query: %s", query)); + + cmisApi.withQuery(query).assertResultsCount().equals(3); + } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/FacetRangeSearchTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/FacetRangeSearchTest.java index 4dfc0c2b9..5b52698a2 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/FacetRangeSearchTest.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/FacetRangeSearchTest.java @@ -22,9 +22,7 @@ import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.core.Is.is; import static org.testng.Assert.assertEquals; import static org.testng.Assert.assertNull; -import static org.testng.Assert.assertTrue; -import java.util.ArrayList; import java.util.List; import java.util.Map; @@ -66,8 +64,8 @@ import org.testng.annotations.Test; * "gap": "+100DAY" * } * } - * @author Michael Suzuki * + * @author Michael Suzuki */ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { @@ -78,49 +76,37 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest waitForContentIndexing(file4.getContent(), true); } + /** Check the error messages mention the mandatory fields when they are omitted. */ @Test @TestRail(section = {TestGroup.REST_API, TestGroup.SEARCH }, executionType = ExecutionType.REGRESSION, description = "Check facet intervals mandatory fields") public void checkingFacetsMandatoryErrorMessages() { SearchRequest query = createQuery("cars"); - List ranges = new ArrayList<>(); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - ranges.add(facetRangeModel); - query.setRanges(ranges); + + // Omit the field. + query.setRanges(List.of(createRangesModel(null, "0", "400", "20"))); query(query); - restClient.assertStatusCodeIs(HttpStatus.BAD_REQUEST).assertLastError() - .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "field")); - ranges.clear(); - facetRangeModel.setField("content.size"); - ranges.add(facetRangeModel); - query.setRanges(ranges); + .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "field")); + // Omit the start. + query.setRanges(List.of(createRangesModel("content.size", null, "400", "20"))); query(query); - restClient.assertStatusCodeIs(HttpStatus.BAD_REQUEST).assertLastError() - .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "start")); - facetRangeModel.setStart("0"); - ranges.clear(); - ranges.add(facetRangeModel); - query.setRanges(ranges); + .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "start")); + // Omit the end. + query.setRanges(List.of(createRangesModel("content.size", "0", null, "20"))); query(query); - restClient.assertStatusCodeIs(HttpStatus.BAD_REQUEST).assertLastError() - .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "end")); - facetRangeModel.setEnd("400"); - query.setRanges(ranges); - ranges.clear(); - ranges.add(facetRangeModel); + .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "end")); + // Omit the gap. + query.setRanges(List.of(createRangesModel("content.size", "0", "400", null))); query(query); - restClient.assertStatusCodeIs(HttpStatus.BAD_REQUEST).assertLastError() - .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "gap")); - - facetRangeModel.setGap("100"); + .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "gap")); } @Test @@ -131,13 +117,8 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { SearchRequest query = createQuery("* AND SITE:'" + testSite.getId() + "'"); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("content.size"); - facetRangeModel.setStart("0"); - facetRangeModel.setEnd("200"); - facetRangeModel.setGap("20"); - List ranges = new ArrayList<>(); - ranges.add(facetRangeModel); + RestRequestRangesModel facetRangeModel = createRangesModel("content.size", "0", "200", "20"); + List ranges = List.of(facetRangeModel); query.setRanges(ranges); SearchResponse response = query(query); response.assertThat().entriesListIsNotEmpty(); @@ -147,14 +128,14 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest RestGenericBucketModel bucket = facetResponseModel.getBuckets().get(0); bucket.assertThat().field("label").is("[20 - 40)"); bucket.assertThat().field("filterQuery").is("content.size:[\"20\" TO \"40\">"); - Map metric = (Map) bucket.getMetrics().get(0).getValue(); + Map metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 2, "Unexpected count for first bucket."); Map info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"20"); - assertEquals(info.get("end"),"40"); + assertEquals(info.get("start"), "20"); + assertEquals(info.get("end"), "40"); assertNull(info.get("count")); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"false"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "false"); bucket = facetResponseModel.getBuckets().get(1); bucket.assertThat().field("label").is("[40 - 120)"); @@ -162,20 +143,20 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for second bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"40"); - assertEquals(info.get("end"),"120"); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"false"); + assertEquals(info.get("start"), "40"); + assertEquals(info.get("end"), "120"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "false"); bucket = facetResponseModel.getBuckets().get(2); bucket.assertThat().field("label").is("[120 - 200]"); bucket.assertThat().field("filterQuery").is("content.size:[\"120\" TO \"200\"]"); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for third bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"120"); - assertEquals(info.get("end"),"200"); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("start"), "120"); + assertEquals(info.get("end"), "200"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "true"); } @Test @@ -186,14 +167,9 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { SearchRequest query = createQuery("* AND SITE:'" + testSite.getId() + "'"); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("content.size"); - facetRangeModel.setStart("0"); - facetRangeModel.setEnd("200"); - facetRangeModel.setGap("20"); + RestRequestRangesModel facetRangeModel = createRangesModel("content.size", "0", "200", "20"); facetRangeModel.setHardend(true); - List ranges = new ArrayList<>(); - ranges.add(facetRangeModel); + List ranges = List.of(facetRangeModel); query.setRanges(ranges); SearchResponse response = query(query); response.assertThat().entriesListIsNotEmpty(); @@ -203,26 +179,26 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest RestGenericBucketModel bucket = facetResponseModel.getBuckets().get(0); bucket.assertThat().field("label").is("[20 - 40)"); bucket.assertThat().field("filterQuery").is("content.size:[\"20\" TO \"40\">"); - Map metric = (Map) bucket.getMetrics().get(0).getValue(); + Map metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 2, "Unexpected count for first bucket."); Map info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"20"); - assertEquals(info.get("end"),"40"); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"false"); + assertEquals(info.get("start"), "20"); + assertEquals(info.get("end"), "40"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "false"); assertNull(info.get("count")); bucket = facetResponseModel.getBuckets().get(1); bucket.assertThat().field("label").is("[40 - 120)"); bucket.assertThat().field("filterQuery").is("content.size:[\"40\" TO \"120\">"); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"40"); - assertEquals(info.get("end"),"120"); + assertEquals(info.get("start"), "40"); + assertEquals(info.get("end"), "120"); metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for second bucket."); assertNull(info.get("count")); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"false"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "false"); bucket = facetResponseModel.getBuckets().get(2); bucket.assertThat().field("label").is("[120 - 200]"); @@ -230,11 +206,11 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for third bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"120"); - assertEquals(info.get("end"),"200"); + assertEquals(info.get("start"), "120"); + assertEquals(info.get("end"), "200"); assertNull(info.get("count")); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "true"); } /** This test relies on a document created in 2015 existing, probably part of the sample site. */ @@ -246,13 +222,8 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { SearchRequest query = createQuery("name:A*"); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("created"); - facetRangeModel.setStart("2015-09-29T10:45:15.729Z"); - facetRangeModel.setEnd("2016-09-29T10:45:15.729Z"); - facetRangeModel.setGap("+280DAY"); - List ranges = new ArrayList<>(); - ranges.add(facetRangeModel); + RestRequestRangesModel facetRangeModel = createRangesModel("created", "2015-09-29T10:45:15.729Z", "2016-09-29T10:45:15.729Z", "+280DAY"); + List ranges = List.of(facetRangeModel); query.setRanges(ranges); SearchResponse response = query(query); response.assertThat().entriesListIsNotEmpty(); @@ -260,18 +231,18 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest RestGenericFacetResponseModel facetResponseModel = response.getContext().getFacets().get(0); List buckets = facetResponseModel.getBuckets(); - assertThat(buckets.size(),is(1)); + assertThat(buckets.size(), is(1)); RestGenericBucketModel bucket = buckets.get(0); bucket.assertThat().field("label").is("[2015-09-29T10:45:15.729Z - 2017-04-11T10:45:15.729Z]"); bucket.assertThat().field("filterQuery").is("created:[\"2015-09-29T10:45:15.729Z\" TO \"2017-04-11T10:45:15.729Z\"]"); bucket.getMetrics().get(0).assertThat().field("value").is("{count=1}"); Map info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"2015-09-29T10:45:15.729Z"); - assertEquals(info.get("end"),"2017-04-11T10:45:15.729Z"); - assertNull(info.get("count"),"1"); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("start"), "2015-09-29T10:45:15.729Z"); + assertEquals(info.get("end"), "2017-04-11T10:45:15.729Z"); + assertNull(info.get("count"), "1"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "true"); } @Test @@ -280,19 +251,9 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest public void searchDateAndSizeRanges() { SearchRequest query = createQuery("* AND SITE:'" + testSite.getId() + "'"); - List ranges = new ArrayList<>(); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("created"); - facetRangeModel.setStart("2015-09-29T10:45:15.729Z"); - facetRangeModel.setEnd("2016-09-29T10:45:15.729Z"); - facetRangeModel.setGap("+280DAY"); - ranges.add(facetRangeModel); - RestRequestRangesModel facetCountRangeModel = new RestRequestRangesModel(); - facetCountRangeModel.setField("content.size"); - facetCountRangeModel.setStart("0"); - facetCountRangeModel.setEnd("500"); - facetCountRangeModel.setGap("200"); - ranges.add(facetCountRangeModel); + RestRequestRangesModel facetRangeModel = createRangesModel("created", "2015-09-29T10:45:15.729Z", "2016-09-29T10:45:15.729Z", "+280DAY"); + RestRequestRangesModel facetCountRangeModel = createRangesModel("content.size", "0", "500", "200"); + List ranges = List.of(facetRangeModel, facetCountRangeModel); query.setRanges(ranges); } @@ -304,16 +265,10 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { SearchRequest query = createQuery("* AND SITE:'" + testSite.getId() + "'"); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("content.size"); - facetRangeModel.setStart("0"); - facetRangeModel.setEnd("200"); - facetRangeModel.setGap("20"); - List include = new ArrayList<>(); - include.add("upper"); + RestRequestRangesModel facetRangeModel = createRangesModel("content.size", "0", "200", "20"); + List include = List.of("upper"); facetRangeModel.setInclude(include); - List ranges = new ArrayList<>(); - ranges.add(facetRangeModel); + List ranges = List.of(facetRangeModel); query.setRanges(ranges); SearchResponse response = query(query); response.assertThat().entriesListIsNotEmpty(); @@ -323,14 +278,14 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest RestGenericBucketModel bucket = facetResponseModel.getBuckets().get(0); bucket.assertThat().field("label").is("(20 - 40]"); bucket.assertThat().field("filterQuery").is("content.size:<\"20\" TO \"40\"]"); - Map metric = (Map) bucket.getMetrics().get(0).getValue(); + Map metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 2, "Unexpected count for first bucket."); Map info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"20"); - assertEquals(info.get("end"),"40"); + assertEquals(info.get("start"), "20"); + assertEquals(info.get("end"), "40"); assertNull(info.get("count")); - assertEquals(info.get("startInclusive"),"false"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("startInclusive"), "false"); + assertEquals(info.get("endInclusive"), "true"); bucket = facetResponseModel.getBuckets().get(1); bucket.assertThat().field("label").is("(40 - 120]"); @@ -338,10 +293,10 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for second bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"40"); - assertEquals(info.get("end"),"120"); - assertEquals(info.get("startInclusive"),"false"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("start"), "40"); + assertEquals(info.get("end"), "120"); + assertEquals(info.get("startInclusive"), "false"); + assertEquals(info.get("endInclusive"), "true"); bucket = facetResponseModel.getBuckets().get(2); bucket.assertThat().field("label").is("(120 - 200]"); @@ -349,9 +304,28 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for third bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"120"); - assertEquals(info.get("end"),"200"); - assertEquals(info.get("startInclusive"),"false"); - assertEquals(info.get("endInclusive"),"true"); - } + assertEquals(info.get("start"), "120"); + assertEquals(info.get("end"), "200"); + assertEquals(info.get("startInclusive"), "false"); + assertEquals(info.get("endInclusive"), "true"); + } + + /** + * Create a ranges model with the values given. + * + * @param field The field to facet on. + * @param start The lowest facet value. + * @param end The highest facet value. + * @param gap The size of the buckets. + * @return The facet ranges model. + */ + private RestRequestRangesModel createRangesModel(String field, String start, String end, String gap) + { + RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); + facetRangeModel.setField(field); + facetRangeModel.setStart(start); + facetRangeModel.setEnd(end); + facetRangeModel.setGap(gap); + return facetRangeModel; + } } \ No newline at end of file diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchAFTSInFieldTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchAFTSInFieldTest.java new file mode 100644 index 000000000..987f141d1 --- /dev/null +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchAFTSInFieldTest.java @@ -0,0 +1,326 @@ +/* + * Copyright 2019 Alfresco Software, Ltd. All rights reserved. + * License rights for this program may be obtained from Alfresco Software, Ltd. + * pursuant to a written agreement and any use of this program without such an + * agreement is prohibited. + */ + +package org.alfresco.test.search.functional.searchServices.search; + +import static java.util.List.of; + +import static jersey.repackaged.com.google.common.collect.Sets.newHashSet; + +import java.util.ArrayList; +import java.util.List; +import java.util.Set; +import java.util.stream.Stream; + +import org.alfresco.rest.model.RestNodeAssociationModelCollection; +import org.alfresco.rest.model.RestNodeChildAssociationModel; +import org.alfresco.rest.search.SearchResponse; +import org.alfresco.search.TestGroup; +import org.alfresco.test.search.functional.AbstractE2EFunctionalTest; +import org.alfresco.utility.model.FileModel; +import org.alfresco.utility.model.FileType; +import org.alfresco.utility.model.FolderModel; +import org.testng.Assert; +import org.testng.annotations.BeforeClass; +import org.testng.annotations.Test; + +/** + * Test class tests AFTS Search In Field works + * Created for Search-840 + * + * @author Meenal Bhave + */ +public class SearchAFTSInFieldTest extends AbstractE2EFunctionalTest +{ + private FolderModel folder1, folder2; + private FileModel file1, file2, file3, file4, file5; + + @BeforeClass(alwaysRun = true) + public void dataPreparation() + { + // Create Folders: + // Folder1: Expected to be found with file.txt + folder1 = new FolderModel("file txt folder"); + dataContent.usingUser(testUser).usingSite(testSite).createFolder(folder1); + + // Folder2: Not expected to be found with file.txt + folder2 = new FolderModel("txt files folder"); + dataContent.usingUser(testUser).usingSite(testSite).createFolder(folder2); + + // Create File(s): Expected to be found with file.txt + file1 = new FileModel("file.txt", "file.txt", "", FileType.TEXT_PLAIN, "file.txt"); + + file2 = new FileModel("1-file.txt", "1-file.txt", "", FileType.TEXT_PLAIN, "1-file.txt"); + + file3 = new FileModel("file1.txt", "file1.txt", "", FileType.TEXT_PLAIN, "file1.txt"); + + file4 = new FileModel("txt file", "txt file", "", FileType.TEXT_PLAIN, "txt file"); + + // Not Expected to be found with file.txt + file5 = new FileModel("txt files", "txt files", "", FileType.TEXT_PLAIN, "txt files"); + + of(file1, file2, file3, file4, file5).forEach( + f -> dataContent.usingUser(testUser).usingSite(testSite).usingResource(folder1).createContent(f)); + + waitForContentIndexing(file5.getContent(), true); + } + + @Test(priority = 1, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldName() + { + // Field names in various formats + Stream fieldNames = Stream.of("{http://www.alfresco.org/model/content/1.0}name", + "@{http://www.alfresco.org/model/content/1.0}name", + "cm_name", + "cm:name", + "@cm:name", + "name"); + + // For each field name, check that queries return consistent results with / out '' + fieldNames.forEach(fieldName -> + { + // Query string without quotes + String query = fieldName + ":file.txt"; + + Set expectedNames = newHashSet(); + expectedNames.add("file.txt"); // file1 + expectedNames.add("1-file.txt"); // file2 + expectedNames.add("file1.txt"); // file3 + expectedNames.add("txt file"); // file4 + expectedNames.add("file txt folder"); // folder1 + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + // Query string in single quotes + query = fieldName + ":'file.txt'"; + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + }); + } + + @Test(priority = 2, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldTitle() + { + // Field names in various formats + Stream fieldNames = Stream.of("{http://www.alfresco.org/model/content/1.0}title", + "@{http://www.alfresco.org/model/content/1.0}title", + "cm_title", + "cm:title", + "@cm:title"); + + // For each field name, check that queries return consistent results with / out '' + fieldNames.forEach(fieldName -> + { + String query = fieldName + ":" + file2.getName(); + boolean fileFound = isContentInSearchResults(query, file2.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + testSearchQuery(query, 1, SearchLanguage.AFTS); + + query = fieldName + ":'" + file2.getName() + "\'"; + fileFound = isContentInSearchResults(query, file2.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + testSearchQuery(query, 1, SearchLanguage.AFTS); + }); + } + + @Test(priority = 3, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldContent() + { + // Field names in various formats + List fieldNames = new ArrayList<>(); + fieldNames.add("TEXT"); + fieldNames.add("{http://www.alfresco.org/model/dictionary/1.0}content"); + fieldNames.add("cm:content"); + fieldNames.add("d:content"); + + // For each field name, check that queries return consistent results with / out '' + fieldNames.forEach(fieldName -> + { + String query = fieldName + ":" + file3.getContent(); + boolean fileFound = isContentInSearchResults(query, file3.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + Integer resultCount1 = testSearchQuery(query, null, SearchLanguage.AFTS).getPagination().getTotalItems(); + + query = fieldName + ":'" + file3.getContent() + "\'"; + fileFound = isContentInSearchResults(query, file3.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + testSearchQuery(query, resultCount1, SearchLanguage.AFTS).getPagination().getTotalItems(); + }); + } + + @Test(priority = 4) + public void testSearchInFieldSITE() + { + String query = "SITE:" + testSite.getId(); + boolean fileFound = isContentInSearchResults(query, folder1.getName(), true); + Assert.assertTrue(fileFound, "Site Not found for query: " + query); + + Integer resultCount1 = testSearchQuery(query, 8, SearchLanguage.AFTS).getPagination().getTotalItems(); + + query = "SITE:'" + testSite.getId() + "\'"; + fileFound = isContentInSearchResults(query, folder1.getName(), true); + Assert.assertTrue(fileFound, "Site Not found for query: " + query); + + testSearchQuery(query, resultCount1, SearchLanguage.AFTS).getPagination().getTotalItems(); + } + + @Test(priority = 5) + public void testSearchInFieldTYPE() + { + // Field names in various formats + List fieldNames = new ArrayList<>(); + fieldNames.add("TYPE"); + fieldNames.add("EXACTTYPE"); + + // For each field name, check that queries return consistent results with / out '' + fieldNames.forEach(fieldName -> { + + String query = fieldName + ":cm\\:content" + " and =cm:name:" + file1.getName(); + boolean fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "Content Not found for query: " + query); + + Integer resultCount1 = testSearchQuery(query, 1, SearchLanguage.AFTS).getPagination().getTotalItems(); + + query = fieldName + ":'cm:content'" + " and =cm:name:" + file1.getName(); + fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "Content Not found for query: " + query); + + testSearchQuery(query, resultCount1, SearchLanguage.AFTS).getPagination().getTotalItems(); + }); + } + + @Test(priority = 6) + public void testSearchInFieldID() throws Exception + { + String query = "ID:'workspace://SpacesStore/" + file1.getNodeRefWithoutVersion() + "'"; + boolean fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "Content Not found for query: " + query); + + testSearchQuery(query, 1, SearchLanguage.AFTS).getPagination().getTotalItems(); + } + + @Test(priority = 7) + public void testSearchInFieldPARENT() + { + String query = "PARENT:" + folder1.getNodeRefWithoutVersion(); + + Set expectedNames = newHashSet(); + expectedNames.add(file1.getName()); + expectedNames.add(file2.getName()); + expectedNames.add(file3.getName()); + expectedNames.add(file4.getName()); + expectedNames.add(file5.getName()); + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + query = "PARENT:'" + folder1.getNodeRefWithoutVersion() + "\'"; + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + } + + @Test(priority = 8) + public void testSearchInFieldPRIMARYPARENT() throws Exception + { + // Create Secondary association in testFolder2 + RestNodeChildAssociationModel childAssoc1 = new RestNodeChildAssociationModel(file1.getNodeRefWithoutVersion(), "cm:contains"); + String secondaryChildrenBody = "[" + childAssoc1.toJson() + "]"; + + restClient.authenticateUser(testUser).withCoreAPI().usingResource(folder2).createSecondaryChildren(secondaryChildrenBody); + RestNodeAssociationModelCollection secondaryChildren = restClient.authenticateUser(testUser).withCoreAPI().usingResource(folder2).getSecondaryChildren(); + secondaryChildren.getEntryByIndex(0).assertThat().field("id").is(file1.getNodeRefWithoutVersion()); + + String query = "PRIMARYPARENT:'workspace://SpacesStore/" + folder1.getNodeRef() + "'"; + + Set expectedNames = newHashSet(); + expectedNames.add(file1.getName()); + expectedNames.add(file2.getName()); + expectedNames.add(file3.getName()); + expectedNames.add(file4.getName()); + expectedNames.add(file5.getName()); + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + query = "PARENT:'workspace://SpacesStore/" + folder2.getNodeRef() + "'"; + boolean fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "Expected Content Not found for query: " + query); + + testSearchQuery(query, 1, SearchLanguage.AFTS); + } + + @Test(priority = 9, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldNameExactMatch() + { + // Check that queries return consistent results with / out '' + String query = "=name:" + file1.getName(); + boolean fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + Integer resultCount1 = testSearchQuery(query, 1, SearchLanguage.AFTS).getPagination().getTotalItems(); + Assert.assertSame(resultCount1, 1, "File count does not match for query: " + query); + + query = "=name:'" + file1.getName() + "\'"; + fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + testSearchQuery(query, resultCount1, SearchLanguage.AFTS).getPagination().getTotalItems(); + } + + @Test(priority = 10, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldNameQueryExpansion() + { + // Check that queries return consistent results with / out '' + String query = "~name:" + file1.getName(); + + Set expectedNames = newHashSet(); + expectedNames.add(file1.getName()); + expectedNames.add(file2.getName()); + expectedNames.add(file3.getName()); + expectedNames.add(file4.getName()); + expectedNames.add(folder1.getName()); + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + query = "~name:'" + file1.getName() + "\'"; + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + } + + @Test(priority = 11, groups = { TestGroup.ACS_63n }) + public void testWithConjunctionDisjunctionAndNegation() + { + // Query string to include Conjunction, Disjunction and Negation + + String query1 = "~name:" + file1.getName(); // Query expected to return 5 results + String query2 = "=name:" + file2.getName(); // Query expected to return 1 result + String query3 = "=name:" + file3.getName(); // Query expected to return 1 result + + // Check Query with Conjunction, Negation, Disjunction: returns right results + // "~name:file.txt and ! (=name:1-file.txt or =name:file1.txt)" + String query = query1 + " and ! (" + query2 + " or " + query3 + ")"; + + // Check that expected files are included in the results + Set expectedNames = newHashSet(); + expectedNames.add("file.txt"); // file1 + expectedNames.add("txt file"); // file4 + expectedNames.add("file txt folder"); // folder1 + + SearchResponse response = testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + // Check result count is 5-(1+1)=3 + int resultCount = response.getPagination().getTotalItems(); + Assert.assertEquals(resultCount, 3, "File count does not match for query: " + query); + + // Check that file2 and file3 are excluded from the results + boolean fileFound = isContentInSearchResponse(response, file2.getName()); + Assert.assertFalse(fileFound, "File2 found for query: " + query); + + fileFound = isContentInSearchResponse(response, file3.getName()); + Assert.assertFalse(fileFound, "File3 found for query: " + query); + } +} diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchSecondaryAssociationTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchSecondaryAssociationTest.java new file mode 100644 index 000000000..b382bc6f3 --- /dev/null +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchSecondaryAssociationTest.java @@ -0,0 +1,91 @@ +/* + * Copyright 2019 Alfresco Software, Ltd. All rights reserved. + * License rights for this program may be obtained from Alfresco Software, Ltd. + * pursuant to a written agreement and any use of this program without such an + * agreement is prohibited. + */ + +package org.alfresco.test.search.functional.searchServices.search; + +import org.alfresco.rest.model.RestNodeAssociationModelCollection; +import org.alfresco.rest.model.RestNodeChildAssociationModel; +import org.alfresco.test.search.functional.AbstractE2EFunctionalTest; +import org.alfresco.utility.data.CustomObjectTypeProperties; +import org.alfresco.utility.model.FileModel; +import org.alfresco.utility.model.FolderModel; +import org.testng.Assert; +import org.testng.annotations.BeforeClass; +import org.testng.annotations.Test; + +/** + * Test class tests content in the secondary parent is found too + * Created for Search-1313 + * + * @author Meenal Bhave + */ +public class SearchSecondaryAssociationTest extends AbstractE2EFunctionalTest +{ + private FolderModel testFolder1, testFolder2; + private FileModel file1; + + @BeforeClass(alwaysRun = true) + public void dataPreparation() + { + // Folders + testFolder1 = new FolderModel("folder1"); + testFolder2 = new FolderModel("folder2"); + + // File(s) + file1 = new FileModel("file1.txt"); + file1.setContent("content file 1"); + + // Create folder1 + dataContent.usingUser(testUser).usingSite(testSite).createCustomContent(testFolder1, "cmis:folder", new CustomObjectTypeProperties()); + + // Create file1 + dataContent.usingUser(testUser).usingResource(testFolder1).createCustomContent(file1, "cmis:document", new CustomObjectTypeProperties()); + + // Create folder2 + dataContent.usingUser(testUser).usingSite(testSite).createCustomContent(testFolder2, "cmis:folder", new CustomObjectTypeProperties()); + + // wait for solr index + waitForMetadataIndexing(file1.getName(), true); + } + + @Test(priority = 1) + public void testSearchPathForSecondaryAssociation() throws Exception + { + String queryPathFolder1 = "PATH:\"/app:company_home/st:sites/cm:" + testSite.getTitle() + + "/cm:documentLibrary/cm:" + testFolder1.getName() + "/cm:" + file1.getName() + "\""; + + // Test if file can be found in folder1: Primary Parent + boolean found = isContentInSearchResults(queryPathFolder1, file1.getName(), true); + Assert.assertTrue(found, "File Not found using Primary Parent Path"); + + String queryPathFolder2 = "PATH:\"/app:company_home/st:sites/cm:" + testSite.getTitle() + + "/cm:documentLibrary/cm:" + testFolder2.getName() + "/cm:" + file1.getName() + "\""; + + // Test if file can not be found in folder2 + found = isContentInSearchResults(queryPathFolder2, file1.getName(), false); + Assert.assertTrue(found, "File found using Secondary Parent Path"); + + // Create Secondary association in folder2 + RestNodeChildAssociationModel childAssoc1 = new RestNodeChildAssociationModel(file1.getNodeRefWithoutVersion(), "cm:contains"); + String secondaryChildrenBody = "[" + childAssoc1.toJson() + "]"; + + restClient.authenticateUser(testUser).withCoreAPI().usingResource(testFolder2).createSecondaryChildren(secondaryChildrenBody); + RestNodeAssociationModelCollection secondaryChildren = restClient.authenticateUser(testUser).withCoreAPI().usingResource(testFolder2).getSecondaryChildren(); + secondaryChildren.getEntryByIndex(0).assertThat().field("id").is(file1.getNodeRefWithoutVersion()); + + // Test if file can be found in folder2: Secondary Parent + found = isContentInSearchResults(queryPathFolder2, file1.getName(), true); + Assert.assertTrue(found, "File Not found using Secondary Parent Path"); + + // Remove Secondary association + restClient.authenticateUser(testUser).withCoreAPI().usingResource(testFolder2).deleteSecondaryChild(secondaryChildren.getEntryByIndex(0)); + + // Test if file can not be found in folder2 + found = isContentInSearchResults(queryPathFolder2, file1.getName(), false); + Assert.assertTrue(found, "File found using Secondary Parent Path"); + } +} diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/ShardInfoTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/ShardInfoTest.java index aac3a485d..087641391 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/ShardInfoTest.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/ShardInfoTest.java @@ -101,6 +101,7 @@ public class ShardInfoTest extends AbstractE2EFunctionalTest RestShardInfoModel model = shardInfoModel.getModel(); assertEquals(model.getTemplate(), "rerank"); assertEquals(model.getShardMethod(), "DB_ID"); + assertEquals(model.getMode(), "MIXED"); assertTrue(model.getHasContent()); assertTrue(stores.contains(model.getStores())); diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/tracker/CascadingTrackerIntegrationTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/tracker/CascadingIntegrationTest.java similarity index 99% rename from e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/tracker/CascadingTrackerIntegrationTest.java rename to e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/tracker/CascadingIntegrationTest.java index 3d66f1922..a61e2dc63 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/tracker/CascadingTrackerIntegrationTest.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/tracker/CascadingIntegrationTest.java @@ -31,7 +31,7 @@ import static org.testng.Assert.assertTrue; * @author Alessandro Benedetti * @author Meenal Bhave */ -public class CascadingTrackerIntegrationTest extends AbstractE2EFunctionalTest +public class CascadingIntegrationTest extends AbstractE2EFunctionalTest { @Autowired protected DataContent dataContent; diff --git a/e2e-test/src/test/resources/testdata/search-by-property.xml b/e2e-test/src/test/resources/testdata/search-by-property.xml deleted file mode 100644 index 0b845ec67..000000000 --- a/e2e-test/src/test/resources/testdata/search-by-property.xml +++ /dev/null @@ -1,188 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/pom.xml b/pom.xml index 8863b968c..f21eb44c8 100644 --- a/pom.xml +++ b/pom.xml @@ -29,7 +29,7 @@ 11 6.6.5 - ${solr.base.version}-patched.1 + ${solr.base.version}-patched.2 @@ -66,6 +66,9 @@ + + alphabetical + diff --git a/search-services/README.md b/search-services/README.md index f44e980ca..0c82e56d1 100644 --- a/search-services/README.md +++ b/search-services/README.md @@ -338,6 +338,22 @@ This Docker Image is available at Alfresco Docker Hub: To use the public image instead of the local one (`searchservices:develop`) just use `alfresco/alfresco-search-services:1.3.x.x` labels. +## Docker Master-Slave setup +### Enable Search Slave Replica config + +To enable slave node specify environment value `REPLICATION_TYPE=slave`, by default Master config is enabled and slave is disabled. + +During deployment time whenever Search Services or Insight Engine image starts, it will execute the script [search_config_setup.sh](/packaging/src/docker) which will configure the slave config setup based on the value specified in the script. + +To run the docker image: + +```bash +$ docker run -p 8984:8983 -e REPLICATION_TYPE=slave -e ALFRESCO_SECURE_COMMS=none -e SOLR_CREATE_ALFRESCO_DEFAULTS=alfresco,archive searchservices:develop +``` +Solr-slave End point: [http://localhost:8984/solr](http://localhost:8984/solr) + +To generate your own Docker-compose file please follow [generator-alfresco-docker-compose](../e2e-test/generator-alfresco-docker-compose/README.md) + ### Use Alfresco Search Services Docker Image with Docker Compose Sample configuration in a Docker Compose file using **Plain HTTP** protocol to communicate with Alfresco Repository. diff --git a/search-services/alfresco-search/doc/architecture/trackers/00001-content-tracker.md b/search-services/alfresco-search/doc/architecture/trackers/00001-content-tracker.md index e267a79a9..0387374eb 100644 --- a/search-services/alfresco-search/doc/architecture/trackers/00001-content-tracker.md +++ b/search-services/alfresco-search/doc/architecture/trackers/00001-content-tracker.md @@ -155,7 +155,6 @@ The following table illustrates the configuration properties used by the Tracker |shard.method|"DB_ID"|Data (Documents, ACLs) Routing criteria among shards| | |Y|Y| | | |alfresco.fingerprint|true|true if we want to compute the content Fingerprint| | |Y|| | | |alfresco.index.transformContent|true| | | |Y|| | | -|alfresco.version|5.0.0|The target Alfresco version| | | | | | | |alfresco.corePoolSize|4|The number of threads to keep in the pool, even if they are idle|Y|Y|Y|Y|Y|Y| |alfresco.maximumPoolSize|-1|The maximum number of threads allowed in the pool|Y|Y|Y|Y|Y|Y| |alfresco.keepAliveTime|120|When the number of threads is greater than the core pool size, this is the maximum time that excess idle threads will wait for new tasks before terminating|Y|Y|Y|Y|Y|Y| diff --git a/search-services/alfresco-search/pom.xml b/search-services/alfresco-search/pom.xml index 8cdd7d1f7..2e37b62de 100644 --- a/search-services/alfresco-search/pom.xml +++ b/search-services/alfresco-search/pom.xml @@ -96,18 +96,79 @@ 2.3.2 + + org.apache.commons + commons-lang3 + 3.9 + + + org.apache.cxf + cxf-core + ${cxf.version} + + + org.apache.cxf + cxf-rt-bindings-soap + ${cxf.version} + + + org.apache.cxf + cxf-rt-bindings-xml + ${cxf.version} + + + org.apache.cxf + cxf-rt-databinding-jaxb + ${cxf.version} + + + org.apache.cxf + cxf-rt-frontend-jaxws + ${cxf.version} + + + org.apache.cxf + cxf-rt-frontend-simple + ${cxf.version} + + + org.apache.cxf + cxf-rt-transports-http + ${cxf.version} + + + org.apache.cxf + cxf-rt-ws-addr + ${cxf.version} + + + org.apache.cxf + cxf-rt-ws-policy + ${cxf.version} + + + org.apache.cxf + cxf-rt-wsdl + ${cxf.version} + + + xpp3 + xpp3 + 1.1.4c + + junit junit - 4.12 + 4.13 test org.mockito mockito-core - 3.1.0 + 3.2.4 test @@ -120,7 +181,7 @@ com.carrotsearch.randomizedtesting randomizedtesting-runner - 2.7.4 + 2.7.6 test @@ -131,6 +192,43 @@ + + + alfresco-public + https://artifacts.alfresco.com/nexus/content/groups/public + + true + + + false + + + + alfresco-public-snapshots + https://artifacts.alfresco.com/nexus/content/groups/public-snapshots + + false + + + true + + + + central + Central Repository + https://repo.maven.apache.org/maven2 + default + + false + + + + maven-restlet + Public online Restlet repository + http://maven.restlet.talend.com + + + alfresco-solr diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableMLTextFieldType.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableMLTextFieldType.java index 03d31c672..89d171642 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableMLTextFieldType.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableMLTextFieldType.java @@ -39,12 +39,9 @@ import org.springframework.extensions.surf.util.I18NUtil; /** * @author Andy - * */ public class AlfrescoCollatableMLTextFieldType extends StrField { - - /* (non-Javadoc) * @see org.apache.solr.schema.StrField#getSortField(org.apache.solr.schema.SchemaField, boolean) */ @@ -75,7 +72,6 @@ public class AlfrescoCollatableMLTextFieldType extends StrField } - public static class MLTextSortFieldComparatorSource extends FieldComparatorSource { @@ -101,16 +97,20 @@ public class AlfrescoCollatableMLTextFieldType extends StrField private final String[] values; private BinaryDocValues docTerms; - - private Bits docsWithField; + + /** + * An array of flags - one for each document in the segment. Each bit is set to true if the document has the + * field or false otherwise. If this is set to null then all docs in the segment have the field. + */ + Bits docsWithField; private final String field; - final Collator collator; + Collator collator; - private String bottom; - - private String top; + String bottom; + + String top; Locale collatorLocale; @@ -138,7 +138,7 @@ public class AlfrescoCollatableMLTextFieldType extends StrField { final String comparableString = findBestValue(doc, docTerms.get(doc)); return compareValues(bottom, comparableString); - + } public void copy(int slot, int doc) @@ -153,13 +153,14 @@ public class AlfrescoCollatableMLTextFieldType extends StrField private String findBestValue(int doc, BytesRef term) { - if (term.length == 0 && docsWithField.get(doc) == false) { + if (term.length == 0 && docsWithField != null && docsWithField.get(doc) == false) + { return null; } - + String withLocale = term.utf8ToString(); - - // split strin into MLText object + + // split string into MLText object if (withLocale == null) { return withLocale; @@ -231,14 +232,15 @@ public class AlfrescoCollatableMLTextFieldType extends StrField { docTerms = DocValues.getBinary(context.reader(), field); docsWithField = DocValues.getDocsWithField(context.reader(), field); - if (docsWithField instanceof Bits.MatchAllBits) { - docsWithField = null; + if (docsWithField instanceof Bits.MatchAllBits) + { + docsWithField = null; } return this; } - + @Override - public int compareValues(String val1, String val2) + public int compareValues(String val1, String val2) { if (val1 == null) { @@ -254,9 +256,10 @@ public class AlfrescoCollatableMLTextFieldType extends StrField } return collator.compare(val1, val2); } - - @Override - public void setScorer(Scorer scorer) {} - } + @Override + public void setScorer(Scorer scorer) + { + } + } } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableTextFieldType.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableTextFieldType.java index d3d54cd63..604dbcc27 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableTextFieldType.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableTextFieldType.java @@ -104,16 +104,20 @@ public class AlfrescoCollatableTextFieldType extends StrField private final String[] values; private BinaryDocValues docTerms; - - private Bits docsWithField; + + /** + * An array of flags - one for each document in the segment. Each bit is set to true if the document has the + * field or false otherwise. If this is set to null then all docs in the segment have the field. + */ + Bits docsWithField; private final String field; - final Collator collator; + Collator collator; - private String bottom; + String bottom; - private String top; + String top; Locale collatorLocale; @@ -141,7 +145,6 @@ public class AlfrescoCollatableTextFieldType extends StrField { final String comparableString = findBestValue(doc, docTerms.get(doc)); return compareValues(bottom, comparableString); - } public void copy(int slot, int doc) @@ -156,7 +159,8 @@ public class AlfrescoCollatableTextFieldType extends StrField private String findBestValue(int doc, BytesRef term) { - if (term.length == 0 && docsWithField.get(doc) == false) { + if (term.length == 0 && docsWithField != null && docsWithField.get(doc) == false) + { return null; } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoSolrDataModel.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoSolrDataModel.java index c28d7ae8b..50b6d9563 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoSolrDataModel.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoSolrDataModel.java @@ -523,7 +523,7 @@ public class AlfrescoSolrDataModel implements QueryConstants } catch (IOException e) { - log.info("Failed to read shared properties fat " + propertiesFile.getAbsolutePath()); + log.info("Failed to read shared properties at " + propertiesFile.getAbsolutePath()); } return props; diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/HandlerReportHelper.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/HandlerReportHelper.java index a3c2b069d..72c3e27c0 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/HandlerReportHelper.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/HandlerReportHelper.java @@ -145,9 +145,6 @@ class HandlerReportHelper return payload; } - /** - * Builds Tracker report - */ static NamedList buildTrackerReport(TrackerRegistry trackerRegistry, InformationServer srv, String coreName, Long fromTx, Long toTx, Long fromAclTx, Long toAclTx, Long fromTime, Long toTime) throws JSONException { @@ -157,7 +154,6 @@ class HandlerReportHelper AclTracker aclTracker = trackerRegistry.getTrackerForCore(coreName, AclTracker.class); IndexHealthReport aclReport = aclTracker.checkIndex(toTx, toAclTx, fromTime, toTime); NamedList ihr = new SimpleOrderedMap<>(); - ihr.add("Alfresco version", aclTracker.getAlfrescoVersion()); ihr.add("DB acl transaction count", aclReport.getDbAclTransactionCount()); ihr.add("Count of duplicated acl transactions in the index", aclReport.getDuplicatedAclTxInIndex() .cardinality()); @@ -277,12 +273,6 @@ class HandlerReportHelper Date end = new Date(now.getTime() + remainingTxTimeMillis); Duration remainingTx = new Duration(now, end); - long remainingChangeSetTimeMillis = 0; - - now = new Date(); - end = new Date(now.getTime() + remainingChangeSetTimeMillis); - Duration remainingChangeSet = new Duration(now, end); - NamedList ftsSummary = new SimpleOrderedMap<>(); long remainingContentTimeMillis = 0; srv.addFTSStatusCounts(ftsSummary); diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/InformationServer.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/InformationServer.java index ec64e11b6..e1b24d82b 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/InformationServer.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/InformationServer.java @@ -88,7 +88,7 @@ public interface InformationServer extends InformationServerCollectionProvider void indexNode(Node node, boolean overwrite) throws IOException, AuthenticationException, JSONException; - void indexNodes(List nodes, boolean overwrite, boolean cascade) throws IOException, AuthenticationException, JSONException; + void indexNodes(List nodes, boolean overwrite) throws IOException, AuthenticationException, JSONException; void cascadeNodes(List nodes, boolean overwrite) throws IOException, AuthenticationException, JSONException; @@ -181,4 +181,11 @@ public interface InformationServer extends InformationServerCollectionProvider String getBaseUrl(); void flushContentStore() throws IOException; + + /** + * Check if cascade tracking is enabled. + * + * @return true if cascade tracking is enabled (note that this is the default behaviour if not specified in the properties file). + */ + boolean cascadeTrackingEnabled(); } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java index 189a5cf8f..e88a75d31 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2015 Alfresco Software Limited. + * Copyright (C) 2019 Alfresco Software Limited. * * This file is part of Alfresco * @@ -19,6 +19,7 @@ package org.alfresco.solr; import static java.util.Optional.ofNullable; + import static org.alfresco.repo.search.adaptor.lucene.QueryConstants.FIELD_ACLID; import static org.alfresco.repo.search.adaptor.lucene.QueryConstants.FIELD_ACLTXCOMMITTIME; import static org.alfresco.repo.search.adaptor.lucene.QueryConstants.FIELD_ACLTXID; @@ -76,16 +77,32 @@ import java.io.PrintWriter; import java.io.StringWriter; import java.nio.charset.StandardCharsets; import java.text.DecimalFormat; -import java.util.*; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.EnumMap; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Locale; +import java.util.Map; import java.util.Map.Entry; +import java.util.Objects; +import java.util.Properties; +import java.util.Set; import java.util.concurrent.locks.ReentrantReadWriteLock; import java.util.regex.Matcher; import java.util.regex.Pattern; +import java.util.zip.GZIPInputStream; import com.carrotsearch.hppc.IntArrayList; - import com.carrotsearch.hppc.LongHashSet; import com.carrotsearch.hppc.cursors.LongCursor; + import org.alfresco.httpclient.AuthenticationException; import org.alfresco.model.ContentModel; import org.alfresco.opencmis.dictionary.CMISStrictDictionaryService; @@ -134,13 +151,26 @@ import org.apache.lucene.analysis.Analyzer; import org.apache.lucene.analysis.TokenStream; import org.apache.lucene.analysis.tokenattributes.CharTermAttribute; import org.apache.lucene.document.Document; +import org.apache.lucene.document.LongPoint; import org.apache.lucene.index.IndexCommit; import org.apache.lucene.index.IndexableField; import org.apache.lucene.index.LeafReaderContext; import org.apache.lucene.index.NumericDocValues; import org.apache.lucene.index.ReaderUtil; import org.apache.lucene.index.Term; -import org.apache.lucene.search.*; +import org.apache.lucene.search.BooleanClause; +import org.apache.lucene.search.BooleanQuery; +import org.apache.lucene.search.Collector; +import org.apache.lucene.search.LeafCollector; +import org.apache.lucene.search.LegacyNumericRangeQuery; +import org.apache.lucene.search.Query; +import org.apache.lucene.search.ScoreDoc; +import org.apache.lucene.search.Scorer; +import org.apache.lucene.search.Sort; +import org.apache.lucene.search.SortField; +import org.apache.lucene.search.TermQuery; +import org.apache.lucene.search.TopDocs; +import org.apache.lucene.search.TopFieldCollector; import org.apache.lucene.util.BytesRefBuilder; import org.apache.solr.common.SolrDocument; import org.apache.solr.common.SolrDocumentList; @@ -164,6 +194,7 @@ import org.apache.solr.schema.SchemaField; import org.apache.solr.search.DelegatingCollector; import org.apache.solr.search.DocIterator; import org.apache.solr.search.DocList; +import org.apache.solr.search.DocSet; import org.apache.solr.search.QueryWrapperFilter; import org.apache.solr.search.SolrIndexSearcher; import org.apache.solr.update.AddUpdateCommand; @@ -178,14 +209,14 @@ import org.springframework.extensions.surf.util.I18NUtil; import org.springframework.util.FileCopyUtils; /** - * This is the Solr4 implementation of the information server (index). + * This is the Apache Solr implementation of the information server (index). + * * @author Ahmed Owian * @since 5.0 */ public class SolrInformationServer implements InformationServer { private final static Log LOGGER = new Log(SolrInformationServer.class); - private final static long TWO_MINUTES = 120000; private static final String NO_SITE = "_REPOSITORY_"; private static final String SHARED_FILES = "_SHARED_FILES_"; @@ -200,23 +231,20 @@ public class SolrInformationServer implements InformationServer public static final String AND = " AND "; public static final String OR = " OR "; - public static final String REQUEST_HANDLER_ALFRESCO_FULL_TEXT_SEARCH = "/afts"; - public static final String REQUEST_HANDLER_NATIVE = "/native"; - public static final String REQUEST_HANDLER_ALFRESCO = "/alfresco"; - public static final String REQUEST_HANDLER_SELECT = "/select"; - public static final String REQUEST_HANDLER_GET = "/get"; - public static final String RESPONSE_DEFAULT_IDS = "response"; - public static final String RESPONSE_DEFAULT_ID = "doc"; + private static final String REQUEST_HANDLER_NATIVE = "/native"; + static final String REQUEST_HANDLER_GET = "/get"; + private static final String RESPONSE_DEFAULT_IDS = "response"; + static final String RESPONSE_DEFAULT_ID = "doc"; - public static final String PREFIX_ERROR = "ERROR-"; + static final String PREFIX_ERROR = "ERROR-"; public static final String DOC_TYPE_NODE = "Node"; - public static final String DOC_TYPE_UNINDEXED_NODE = "UnindexedNode"; - public static final String DOC_TYPE_ERROR_NODE = "ErrorNode"; + private static final String DOC_TYPE_UNINDEXED_NODE = "UnindexedNode"; + private static final String DOC_TYPE_ERROR_NODE = "ErrorNode"; public static final String DOC_TYPE_ACL = "Acl"; public static final String DOC_TYPE_TX = "Tx"; public static final String DOC_TYPE_ACL_TX = "AclTx"; - public static final String DOC_TYPE_STATE = "State"; + private static final String DOC_TYPE_STATE = "State"; public static final String SOLR_HOST = "solr.host"; public static final String SOLR_PORT = "solr.port"; @@ -227,6 +255,8 @@ public class SolrInformationServer implements InformationServer */ private static final int BATCH_FACET_TXS = 4096; private static final String FINGERPRINT_FIELD = "MINHASH"; + /** Shared property to determine if the cascade tracking is enabled. */ + public static final String CASCADE_TRACKER_ENABLED = "alfresco.cascade.tracker.enabled"; private final AlfrescoCoreAdminHandler adminHandler; private final SolrCore core; @@ -235,7 +265,6 @@ public class SolrInformationServer implements InformationServer private final TrackerStats trackerStats = new TrackerStats(this); private final AlfrescoSolrDataModel dataModel; private final SolrContentStore solrContentStore; - private final String alfrescoVersion; private final boolean transformContent; private final boolean recordUnindexedNodes; private final long lag; @@ -271,7 +300,7 @@ public class SolrInformationServer implements InformationServer protected enum FTSStatus {New, Dirty, Clean} - class DocListCollector implements Collector, LeafCollector + static class DocListCollector implements Collector, LeafCollector { private IntArrayList docs = new IntArrayList(); private int docBase; @@ -303,7 +332,7 @@ public class SolrInformationServer implements InformationServer } } - class TxnCacheFilter extends DelegatingCollector + static class TxnCacheFilter extends DelegatingCollector { private NumericDocValues currentLongs; private Map txnLRU; @@ -330,7 +359,7 @@ public class SolrInformationServer implements InformationServer } } - class TxnCollector extends DelegatingCollector + static class TxnCollector extends DelegatingCollector { private NumericDocValues currentLongs; private long txnFloor; @@ -372,7 +401,7 @@ public class SolrInformationServer implements InformationServer } } - class LRU extends LinkedHashMap + static class LRU extends LinkedHashMap { private int maxSize; @@ -395,7 +424,7 @@ public class SolrInformationServer implements InformationServer boolean isDefinitionExists(QName qName); } - abstract class TransactionInfoReporter + static abstract class TransactionInfoReporter { protected final IndexHealthReport report; @@ -428,7 +457,6 @@ public class SolrInformationServer implements InformationServer this.solrContentStore = solrContentStore; Properties p = core.getResourceLoader().getCoreProperties(); - alfrescoVersion = p.getProperty("alfresco.version", "Unknown"); transformContent = Boolean.parseBoolean(p.getProperty("alfresco.index.transformContent", "true")); recordUnindexedNodes = Boolean.parseBoolean(p.getProperty("alfresco.recordUnindexedNodes", "true")); lag = Integer.parseInt(p.getProperty("alfresco.lag", "1000")); @@ -438,7 +466,7 @@ public class SolrInformationServer implements InformationServer dataModel = AlfrescoSolrDataModel.getInstance(); contentStreamLimit = Integer.parseInt(p.getProperty("alfresco.contentStreamLimit", "10000000")); - + // build base URL - host and port have to come from configuration. props = AlfrescoSolrDataModel.getCommonConfig(); hostName = ConfigUtil.locateProperty(SOLR_HOST, props.getProperty(SOLR_HOST)); @@ -453,6 +481,14 @@ public class SolrInformationServer implements InformationServer return this.adminHandler; } + /** {@inheritDoc} */ + @Override + public boolean cascadeTrackingEnabled() + { + String cascadeTrackerEnabledProp = ofNullable((String) props.get(CASCADE_TRACKER_ENABLED)).orElse("true"); + return Boolean.valueOf(cascadeTrackerEnabledProp); + } + @Override public synchronized void initSkippingDescendantDocs() { @@ -504,12 +540,7 @@ public class SolrInformationServer implements InformationServer report.add("Node count with FTSStatus New", newCount); } } - - public String getAlfrescoVersion() - { - return this.alfrescoVersion; - } - + @Override public void afterInitModels() { @@ -522,7 +553,7 @@ public class SolrInformationServer implements InformationServer String query = FIELD_ACLID + ":" + aclid + AND + FIELD_DOC_TYPE + ":" + DOC_TYPE_ACL; long count = this.getDocListSize(query); aclReport.setIndexedAclDocCount(count); - + // TODO Could add INACLTXID later, but would need acl change set id. return aclReport; } @@ -780,7 +811,7 @@ public class SolrInformationServer implements InformationServer long count = this.getDocListSize(query); nodeReport.setIndexedNodeDocCount(count); } - + @Override public void commit() throws IOException { @@ -893,26 +924,26 @@ public class SolrInformationServer implements InformationServer return searcherOpened; } - + @Override public void deleteByAclChangeSetId(Long aclChangeSetId) throws IOException { deleteById(FIELD_INACLTXID, aclChangeSetId); } - + @Override public void deleteByAclId(Long aclId) throws IOException { isIdIndexCache.clear(); deleteById(FIELD_ACLID, aclId); } - + @Override public void deleteByNodeId(Long nodeId) throws IOException { deleteById(FIELD_DBID, nodeId); } - + @Override public void deleteByTransactionId(Long transactionId) throws IOException { @@ -925,7 +956,7 @@ public class SolrInformationServer implements InformationServer { return this.dataModel.getAlfrescoModels(); } - + @SuppressWarnings({ "unchecked", "rawtypes" }) @Override public Iterable> getCoreStats() throws IOException @@ -1049,7 +1080,7 @@ public class SolrInformationServer implements InformationServer return coreSummary; } - + @Override public DictionaryComponent getDictionaryService(String alternativeDictionary) { @@ -1373,7 +1404,10 @@ public class SolrInformationServer implements InformationServer public void dirtyTransaction(long txnId) { this.cleanContentCache.remove(txnId); - this.cleanCascadeCache.remove(txnId); + if (cascadeTrackingEnabled()) + { + this.cleanCascadeCache.remove(txnId); + } } @Override @@ -1462,9 +1496,9 @@ public class SolrInformationServer implements InformationServer long start = System.nanoTime(); if ((node.getStatus() == SolrApiNodeStatus.DELETED) - || (node.getStatus() == SolrApiNodeStatus.NON_SHARD_DELETED) - || (node.getStatus() == SolrApiNodeStatus.NON_SHARD_UPDATED) - || (node.getStatus() == SolrApiNodeStatus.UNKNOWN)) + || (node.getStatus() == SolrApiNodeStatus.UNKNOWN) + || cascadeTrackingEnabled() && ((node.getStatus() == SolrApiNodeStatus.NON_SHARD_DELETED) + || (node.getStatus() == SolrApiNodeStatus.NON_SHARD_UPDATED))) { // fix up any secondary paths NodeMetaDataParameters nmdp = new NodeMetaDataParameters(); @@ -1472,8 +1506,8 @@ public class SolrInformationServer implements InformationServer nmdp.setToNodeId(node.getId()); List nodeMetaDatas; if ((node.getStatus() == SolrApiNodeStatus.DELETED) - || (node.getStatus() == SolrApiNodeStatus.NON_SHARD_DELETED) - || (node.getStatus() == SolrApiNodeStatus.NON_SHARD_UPDATED)) + || cascadeTrackingEnabled() && ((node.getStatus() == SolrApiNodeStatus.NON_SHARD_DELETED) + || (node.getStatus() == SolrApiNodeStatus.NON_SHARD_UPDATED))) { // Fake the empty node metadata for this parent deleted node NodeMetaData nodeMetaData = createDeletedNodeMetaData(node); @@ -1483,7 +1517,7 @@ public class SolrInformationServer implements InformationServer { nodeMetaDatas = repositoryClient.getNodesMetaData(nmdp, Integer.MAX_VALUE); } - + NodeMetaData nodeMetaData; if (!nodeMetaDatas.isEmpty()) { @@ -1501,7 +1535,7 @@ public class SolrInformationServer implements InformationServer finally { unlock(nodeMetaData.getId()); - } + } } } // else, the node has moved on to a later transaction, and it will be indexed later @@ -1510,10 +1544,9 @@ public class SolrInformationServer implements InformationServer deleteNode(processor, request, node); } - - if ((node.getStatus() == SolrApiNodeStatus.UPDATED) - || (node.getStatus() == SolrApiNodeStatus.UNKNOWN) - || (node.getStatus() == SolrApiNodeStatus.NON_SHARD_UPDATED)) + if (node.getStatus() == SolrApiNodeStatus.UPDATED + || node.getStatus() == SolrApiNodeStatus.UNKNOWN + || (cascadeTrackingEnabled() && node.getStatus() == SolrApiNodeStatus.NON_SHARD_UPDATED)) { long nodeId = node.getId(); @@ -1547,7 +1580,7 @@ public class SolrInformationServer implements InformationServer StringPropertyValue pValue = (StringPropertyValue) properties.get(ContentModel.PROP_IS_INDEXED); if (pValue != null) { - boolean isIndexed = Boolean.valueOf(pValue.getValue()); + boolean isIndexed = Boolean.parseBoolean(pValue.getValue()); if (!isIndexed) { LOGGER.debug("Clearing unindexed"); @@ -1787,7 +1820,7 @@ public class SolrInformationServer implements InformationServer @Override - public void indexNodes(List nodes, boolean overwrite, boolean cascade) throws IOException, JSONException + public void indexNodes(List nodes, boolean overwrite) throws IOException, JSONException { UpdateRequestProcessor processor = null; try (SolrQueryRequest request = newSolrQueryRequest()) @@ -1798,8 +1831,13 @@ public class SolrInformationServer implements InformationServer EnumMap> nodeStatusToNodeIds = new EnumMap<>(SolrApiNodeStatus.class); categorizeNodes(nodes, nodeIdsToNodes, nodeStatusToNodeIds); List deletedNodeIds = mapNullToEmptyList(nodeStatusToNodeIds.get(SolrApiNodeStatus.DELETED)); - List shardDeletedNodeIds = mapNullToEmptyList(nodeStatusToNodeIds.get(SolrApiNodeStatus.NON_SHARD_DELETED)); - List shardUpdatedNodeIds = mapNullToEmptyList(nodeStatusToNodeIds.get(SolrApiNodeStatus.NON_SHARD_UPDATED)); + List shardDeletedNodeIds = Collections.emptyList(); + List shardUpdatedNodeIds = Collections.emptyList(); + if (cascadeTrackingEnabled()) + { + shardDeletedNodeIds = mapNullToEmptyList(nodeStatusToNodeIds.get(SolrApiNodeStatus.NON_SHARD_DELETED)); + shardUpdatedNodeIds = mapNullToEmptyList(nodeStatusToNodeIds.get(SolrApiNodeStatus.NON_SHARD_UPDATED)); + } List unknownNodeIds = mapNullToEmptyList(nodeStatusToNodeIds.get(SolrApiNodeStatus.UNKNOWN)); List updatedNodeIds = mapNullToEmptyList(nodeStatusToNodeIds.get(SolrApiNodeStatus.UPDATED)); @@ -1820,6 +1858,12 @@ public class SolrInformationServer implements InformationServer { NodeMetaDataParameters nmdp = new NodeMetaDataParameters(); nmdp.setNodeIds(unknownNodeIds); + // When deleting nodes, no additional information is required + nmdp.setIncludeChildIds(false); + nmdp.setIncludeChildAssociations(false); + nmdp.setIncludeAspects(false); + nmdp.setIncludePaths(false); + nmdp.setIncludeParentAssociations(false); nodeMetaDatas.addAll(repositoryClient.getNodesMetaData(nmdp, Integer.MAX_VALUE)); } @@ -1861,10 +1905,12 @@ public class SolrInformationServer implements InformationServer nodeIds.addAll(unknownNodeIds); nodeIds.addAll(shardUpdatedNodeIds); nmdp.setNodeIds(nodeIds); + nmdp.setIncludeChildIds(false); + nmdp.setIncludeChildAssociations(false); // Fetches bulk metadata List nodeMetaDatas = repositoryClient.getNodesMetaData(nmdp, Integer.MAX_VALUE); - + NEXT_NODE: for (NodeMetaData nodeMetaData : nodeMetaDatas) { @@ -1883,7 +1929,7 @@ public class SolrInformationServer implements InformationServer continue; } - if (nodeIdsToNodes.get(nodeMetaData.getId()).getStatus() == SolrApiNodeStatus.NON_SHARD_UPDATED) + if (cascadeTrackingEnabled() && nodeIdsToNodes.get(nodeMetaData.getId()).getStatus() == SolrApiNodeStatus.NON_SHARD_UPDATED) { if (nodeMetaData.getProperties().get(ContentModel.PROP_CASCADE_TX) != null) { @@ -1902,7 +1948,7 @@ public class SolrInformationServer implements InformationServer StringPropertyValue pValue = (StringPropertyValue) properties.get(ContentModel.PROP_IS_INDEXED); if (pValue != null) { - boolean isIndexed = Boolean.valueOf(pValue.getValue()); + boolean isIndexed = Boolean.parseBoolean(pValue.getValue()); if (!isIndexed) { LOGGER.debug("Clearing unindexed"); @@ -1958,7 +2004,7 @@ public class SolrInformationServer implements InformationServer } } - private void addToNewDocAndCache(NodeMetaData nodeMetaData, SolrInputDocument newDoc) throws IOException + private void addToNewDocAndCache(NodeMetaData nodeMetaData, SolrInputDocument newDoc) { addFieldsToDoc(nodeMetaData, newDoc); SolrInputDocument cachedDoc = null; @@ -1984,7 +2030,7 @@ public class SolrInformationServer implements InformationServer { StringPropertyValue latProp = ((StringPropertyValue)nodeMetaData.getProperties().get(ContentModel.PROP_LATITUDE)); StringPropertyValue lonProp = ((StringPropertyValue)nodeMetaData.getProperties().get(ContentModel.PROP_LONGITUDE)); - + if((latProp != null) && (lonProp != null)) { String lat = latProp.getValue(); @@ -2022,14 +2068,14 @@ public class SolrInformationServer implements InformationServer } doc.addField(FIELD_ISNODE, "T"); // FIELD_FTSSTATUS is set when adding content properties to indicate whether or not the cache is clean. - + doc.addField(FIELD_TENANT, AlfrescoSolrDataModel.getTenantId(nodeMetaData.getTenantDomain())); updatePathRelatedFields(nodeMetaData, doc); updateNamePathRelatedFields(nodeMetaData, doc); updateAncestorRelatedFields(nodeMetaData, doc); doc.addField(FIELD_PARENT_ASSOC_CRC, nodeMetaData.getParentAssocsCrc()); - + if (nodeMetaData.getOwner() != null) { doc.addField(FIELD_OWNER, nodeMetaData.getOwner()); @@ -2115,7 +2161,7 @@ public class SolrInformationServer implements InformationServer } } - static void addPropertiesToDoc(Map properties, boolean isContentIndexedForNode, + static void addPropertiesToDoc(Map properties, boolean isContentIndexedForNode, SolrInputDocument newDoc, SolrInputDocument cachedDoc, boolean transformContentFlag) { for (Entry property : properties.entrySet()) @@ -2123,7 +2169,7 @@ public class SolrInformationServer implements InformationServer QName propertyQName = property.getKey(); newDoc.addField(FIELD_PROPERTIES, propertyQName.toString()); newDoc.addField(FIELD_PROPERTIES, propertyQName.getPrefixString()); - + PropertyValue value = property.getValue(); if(value != null) { @@ -2200,12 +2246,20 @@ public class SolrInformationServer implements InformationServer private void deleteErrorNode(UpdateRequestProcessor processor, SolrQueryRequest request, Node node) throws IOException { - String errorDocId = PREFIX_ERROR + node.getId(); - DeleteUpdateCommand delErrorDocCmd = new DeleteUpdateCommand(request); - delErrorDocCmd.setId(errorDocId); - processor.processDelete(delErrorDocCmd); - } + String errorDocId = PREFIX_ERROR + node.getId(); + + // Try finding the node before performing removal operation + DocSet docSet = request.getSearcher().getDocSet(new TermQuery(new Term(FIELD_SOLR4_ID, errorDocId))); + + if (docSet.size() > 0) + { + DeleteUpdateCommand delErrorDocCmd = new DeleteUpdateCommand(request); + delErrorDocCmd.setId(errorDocId); + processor.processDelete(delErrorDocCmd); + } + + } private void deleteNode(UpdateRequestProcessor processor, SolrQueryRequest request, Node node) throws IOException { @@ -2218,9 +2272,17 @@ public class SolrInformationServer implements InformationServer private void deleteNode(UpdateRequestProcessor processor, SolrQueryRequest request, long dbid) throws IOException { - DeleteUpdateCommand delDocCmd = new DeleteUpdateCommand(request); - delDocCmd.setQuery(FIELD_DBID + ":" + dbid); - processor.processDelete(delDocCmd); + + // Try finding the node before performing removal operation + DocSet docSet = request.getSearcher().getDocSet(LongPoint.newExactQuery(FIELD_DBID, dbid)); + + if (docSet.size() > 0) + { + DeleteUpdateCommand delDocCmd = new DeleteUpdateCommand(request); + delDocCmd.setQuery(FIELD_DBID + ":" + dbid); + processor.processDelete(delDocCmd); + } + } private boolean isContentIndexedForNode(Map properties) @@ -2232,7 +2294,7 @@ public class SolrInformationServer implements InformationServer .get(ContentModel.PROP_IS_CONTENT_INDEXED); if (pValue != null) { - boolean isIndexed = Boolean.valueOf(pValue.getValue()); + boolean isIndexed = Boolean.parseBoolean(pValue.getValue()); if (!isIndexed) { isContentIndexed = false; @@ -2282,7 +2344,7 @@ public class SolrInformationServer implements InformationServer return fieldName; } - private void addContentPropertyMetadata(SolrInputDocument doc, QName propertyQName, + private void addContentPropertyMetadata(SolrInputDocument doc, QName propertyQName, AlfrescoSolrDataModel.ContentFieldType type, GetTextContentResponse textContentResponse) { IndexedField indexedField = AlfrescoSolrDataModel.getInstance().getIndexedFieldForContentPropertyMetadata( @@ -2308,7 +2370,7 @@ public class SolrInformationServer implements InformationServer } } - private static void addContentPropertyMetadata(SolrInputDocument doc, QName propertyQName, + private static void addContentPropertyMetadata(SolrInputDocument doc, QName propertyQName, ContentPropertyValue contentPropertyValue, AlfrescoSolrDataModel.ContentFieldType type) { IndexedField indexedField = AlfrescoSolrDataModel.getInstance().getIndexedFieldForContentPropertyMetadata( @@ -2338,8 +2400,8 @@ public class SolrInformationServer implements InformationServer } } } - - private static void addContentPropertyToDocUsingCache(SolrInputDocument newDoc, SolrInputDocument cachedDoc, + + private static void addContentPropertyToDocUsingCache(SolrInputDocument newDoc, SolrInputDocument cachedDoc, QName propertyQName, ContentPropertyValue contentPropertyValue, boolean transformContentFlag) { addContentPropertyMetadata(newDoc, propertyQName, contentPropertyValue, AlfrescoSolrDataModel.ContentFieldType.DOCID); @@ -2347,14 +2409,14 @@ public class SolrInformationServer implements InformationServer addContentPropertyMetadata(newDoc, propertyQName, contentPropertyValue, AlfrescoSolrDataModel.ContentFieldType.LOCALE); addContentPropertyMetadata(newDoc, propertyQName, contentPropertyValue, AlfrescoSolrDataModel.ContentFieldType.MIMETYPE); addContentPropertyMetadata(newDoc, propertyQName, contentPropertyValue, AlfrescoSolrDataModel.ContentFieldType.ENCODING); - + if (!transformContentFlag) { // Marks it as Clean so we do not get the actual content markFTSStatus(newDoc, FTSStatus.Clean); return; } - + if (cachedDoc != null) { ofNullable(cachedDoc.getField("MINHASH")) @@ -2372,32 +2434,38 @@ public class SolrInformationServer implements InformationServer addFieldIfNotSet(newDoc, field); } - String transformationStatusFieldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, + String transformationStatusFieldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, AlfrescoSolrDataModel.ContentFieldType.TRANSFORMATION_STATUS); - newDoc.addField(transformationStatusFieldName, cachedDoc.getFieldValue(transformationStatusFieldName)); + if (transformationStatusFieldName != null){ + newDoc.addField(transformationStatusFieldName, cachedDoc.getFieldValue(transformationStatusFieldName)); + } String transformationExceptionFieldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, AlfrescoSolrDataModel.ContentFieldType.TRANSFORMATION_EXCEPTION); - newDoc.addField(transformationExceptionFieldName, cachedDoc.getFieldValue(transformationExceptionFieldName)); - String transformationTimeFieldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, + if (transformationExceptionFieldName != null){ + newDoc.addField(transformationExceptionFieldName, cachedDoc.getFieldValue(transformationExceptionFieldName)); + } + String transformationTimeFieldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, AlfrescoSolrDataModel.ContentFieldType.TRANSFORMATION_TIME); - newDoc.addField(transformationTimeFieldName, cachedDoc.getFieldValue(transformationTimeFieldName)); + if (transformationTimeFieldName != null){ + newDoc.addField(transformationTimeFieldName, cachedDoc.getFieldValue(transformationTimeFieldName)); + } // Gets the new content docid and compares to that of the cachedDoc to mark the content as clean/dirty String fldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, AlfrescoSolrDataModel.ContentFieldType.DOCID); - + if(newDoc.getFieldValue(FIELD_FTSSTATUS) == null) { newDoc.addField(FIELD_FTSSTATUS, cachedDoc.getFieldValue(FIELD_FTSSTATUS)); } - + if(cachedDoc.getFieldValue(fldName) != null) { - long cachedDocContentDocid = Long.valueOf(String.valueOf(cachedDoc.getFieldValue(fldName))); + long cachedDocContentDocid = Long.parseLong(String.valueOf(cachedDoc.getFieldValue(fldName))); long currentContentDocid = contentPropertyValue.getId(); // If we have used out of date content we mark it as dirty // Otherwise we leave it alone - it could already be marked as dirty/New and require an update - + if (cachedDocContentDocid != currentContentDocid) { // The cached content is out of date @@ -2409,7 +2477,7 @@ public class SolrInformationServer implements InformationServer markFTSStatus(newDoc, FTSStatus.Dirty); } } - else + else { // There is not a SolrInputDocument in the solrContentStore, so no content is added now to the new solr doc markFTSStatus(newDoc, FTSStatus.New); @@ -2448,7 +2516,7 @@ public class SolrInformationServer implements InformationServer doc.removeField(FIELD_FTSSTATUS); doc.addField(FIELD_FTSSTATUS, status.toString()); } - + private void addContentToDoc(SolrInputDocument doc, long dbId) throws AuthenticationException, IOException { Collection fieldNames = doc.deepCopy().getFieldNames(); @@ -2464,15 +2532,15 @@ public class SolrInformationServer implements InformationServer // Could update multi content but it is broken .... } } - + private void addContentPropertyToDocUsingAlfrescoRepository(SolrInputDocument doc, QName propertyQName, long dbId, String locale) throws AuthenticationException, IOException { long start = System.nanoTime(); - + // Expensive call to be done with ContentTracker GetTextContentResponse response = repositoryClient.getTextContent(dbId, propertyQName, null); - + addContentPropertyMetadata(doc, propertyQName, AlfrescoSolrDataModel.ContentFieldType.TRANSFORMATION_STATUS, response); addContentPropertyMetadata(doc, propertyQName, AlfrescoSolrDataModel.ContentFieldType.TRANSFORMATION_EXCEPTION, @@ -2481,6 +2549,10 @@ public class SolrInformationServer implements InformationServer response); InputStream ris = response.getContent(); + if (Objects.equals(response.getContentEncoding(), "gzip")) + { + ris = new GZIPInputStream(ris); + } String textContent = ""; try { @@ -2521,7 +2593,7 @@ public class SolrInformationServer implements InformationServer long end = System.nanoTime(); this.getTrackerStats().addDocTransformationTime(end - start); - + StringBuilder builder = new StringBuilder(textContent.length() + 16); builder.append("\u0000").append(locale).append("\u0000"); builder.append(textContent); @@ -2543,7 +2615,7 @@ public class SolrInformationServer implements InformationServer } private static void addMLTextPropertyToDoc(SolrInputDocument doc, FieldInstance field, MLTextPropertyValue mlTextPropertyValue) - { + { if(field.isLocalised()) { StringBuilder sort = new StringBuilder(128); @@ -2551,20 +2623,20 @@ public class SolrInformationServer implements InformationServer { final String propValue = mlTextPropertyValue.getValue(locale); LOGGER.debug("ML {} in {} of {}", field.getField(), locale, propValue); - + if((locale == null) || (propValue == null)) { continue; - } - + } + StringBuilder builder = new StringBuilder(propValue.length() + 16); builder.append("\u0000").append(locale.toString()).append("\u0000").append(propValue); - + if(!field.isSort()) { doc.addField(field.getField(), builder.toString()); } - + if (sort.length() > 0) { sort.append("\u0000"); @@ -2605,7 +2677,10 @@ public class SolrInformationServer implements InformationServer input.addField(FIELD_INTXID, txn.getId()); input.addField(FIELD_TXCOMMITTIME, txn.getCommitTimeMs()); input.addField(FIELD_DOC_TYPE, DOC_TYPE_TX); - input.addField(FIELD_CASCADE_FLAG, 0); + if (cascadeTrackingEnabled()) + { + input.addField(FIELD_CASCADE_FLAG, 0); + } cmd.solrDoc = input; processor.processAdd(cmd); } @@ -2646,8 +2721,11 @@ public class SolrInformationServer implements InformationServer input.addField(FIELD_S_TXID, info.getId()); input.addField(FIELD_S_TXCOMMITTIME, info.getCommitTimeMs()); - //Set the cascade flag to 1. This means cascading updates have not been done yet. - input.addField(FIELD_CASCADE_FLAG, 1); + if (cascadeTrackingEnabled()) + { + //Set the cascade flag to 1. This means cascading updates have not been done yet. + input.addField(FIELD_CASCADE_FLAG, 1); + } cmd.solrDoc = input; processor.processAdd(cmd); @@ -2874,7 +2952,7 @@ public class SolrInformationServer implements InformationServer { activeTrackerThreadsLock.writeLock().unlock(); } - + } @Override @@ -2901,7 +2979,7 @@ public class SolrInformationServer implements InformationServer SolrIndexSearcher solrIndexSearcher = refCounted.get(); NumericDocValues dbidDocValues = solrIndexSearcher.getSlowAtomicReader().getNumericDocValues(QueryConstants.FIELD_DBID); - + List batch = new ArrayList<>(200); DocList docList = cloud.getDocList(nativeRequestHandler, request, query.startsWith("{") ? query : "{!afts}"+query); for (DocIterator it = docList.iterator(); it.hasNext(); /**/) @@ -2916,16 +2994,16 @@ public class SolrInformationServer implements InformationServer node.setTxnId(Long.MAX_VALUE); batch.add(node); - + if(batch.size() >= 200) { - indexNodes(batch, true, true); + indexNodes(batch, true); batch.clear(); } } if(batch.size() > 0) { - indexNodes(batch, true, true); + indexNodes(batch, true); batch.clear(); } } @@ -3414,7 +3492,7 @@ public class SolrInformationServer implements InformationServer SolrQueryRequest request, UpdateRequestProcessor processor, LinkedHashSet stack) throws AuthenticationException, IOException, JSONException { - + // skipDescendantDocsForSpecificAspects is initialised on a synchronised method, so access must be also synchronised synchronized (this) { @@ -3584,7 +3662,7 @@ public class SolrInformationServer implements InformationServer field, 1); // Min count of 1 ensures that the id returned is in the index for (Map.Entry idCount : idCounts) { - long idInIndex = Long.valueOf(idCount.getKey()); + long idInIndex = Long.parseLong(idCount.getKey()); // Only looks at facet values that fit the query if (batchStartId <= idInIndex && idInIndex <= batchEndId) diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/content/SolrContentStore.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/content/SolrContentStore.java index a11f3f4ba..1a3694a11 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/content/SolrContentStore.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/content/SolrContentStore.java @@ -43,12 +43,14 @@ import java.io.OutputStream; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.nio.file.StandardCopyOption; import java.text.SimpleDateFormat; import java.util.Date; import java.util.List; import java.util.Locale; import java.util.Map; import java.util.function.Predicate; +import java.util.stream.Stream; import java.util.zip.GZIPInputStream; import java.util.zip.GZIPOutputStream; @@ -236,7 +238,8 @@ public final class SolrContentStore implements Closeable, AccessMode { try { - return Files.lines(Paths.get(root, ".version")) + return Files.readAllLines(Paths.get(root, ".version")) + .stream() .map(Long::parseLong) .findFirst() .orElse(NO_VERSION_AVAILABLE); @@ -250,18 +253,29 @@ public final class SolrContentStore implements Closeable, AccessMode @Override public void setLastCommittedVersion(long version) { + + File tmpFile = new File(root, ".version-" + new SimpleDateFormat(SnapShooter.DATE_FMT, Locale.ROOT).format(new Date())); try { - File tmpFile = new File(root, ".version-" + new SimpleDateFormat(SnapShooter.DATE_FMT, Locale.ROOT).format(new Date())); FileWriter wr = new FileWriter(tmpFile); wr.write(Long.toString(version)); wr.close(); - tmpFile.renameTo(new File(root, ".version")); + // file.renameTo(..) does not work on windows. Use Files.move instead. + Files.move(tmpFile.toPath(), new File(root, ".version").toPath(), StandardCopyOption.ATOMIC_MOVE); + } catch (IOException exception) { logger.error("Unable to persist the last committed content store version {}. See the stacktrace below for furtger details.", version, exception); + try + { + Files.delete(tmpFile.toPath()); + } + catch (IOException e) + { + logger.error("Unable to delete tmp contentstore version file {}.", version); + } } } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoIndexFetcher.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoIndexFetcher.java index ee1666778..bb36889e7 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoIndexFetcher.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoIndexFetcher.java @@ -34,40 +34,10 @@ */ package org.alfresco.solr.handler; -import static java.util.List.of; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.ALIAS; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CHECKSUM; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_CONTENT_STORE_FILES; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_DETAILS; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_GET_FILE; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_GET_FILE_LIST; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_INDEX_VERSION; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.COMMAND; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.COMPRESSION; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONF_FILES; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONF_FILE_SHORT; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_FILES; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_FILE_LIST; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_VERSION; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.EXTERNAL; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FILE; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FILE_STREAM; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FileInfo; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.GENERATION; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.INTERNAL; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.MASTER_URL; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.NO_INDEX_REPLICATION_REQUIRED; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.OFFSET; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.SIZE; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.TLOG_FILE; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.TLOG_FILES; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.getCheckSum; -import static org.apache.solr.common.params.CommonParams.JAVABIN; -import static org.apache.solr.common.params.CommonParams.NAME; - import com.google.common.base.Strings; import com.google.common.collect.Lists; import org.alfresco.solr.content.SolrContentStore; +import org.apache.commons.io.FilenameUtils; import org.apache.http.client.HttpClient; import org.apache.lucene.codecs.CodecUtil; import org.apache.lucene.index.IndexCommit; @@ -152,6 +122,37 @@ import java.util.zip.Adler32; import java.util.zip.Checksum; import java.util.zip.InflaterInputStream; +import static java.util.List.of; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.ALIAS; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CHECKSUM; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_CONTENT_STORE_FILES; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_DETAILS; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_GET_FILE; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_GET_FILE_LIST; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_INDEX_VERSION; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.COMMAND; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.COMPRESSION; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONF_FILES; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONF_FILE_SHORT; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_FILES; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_FILE_LIST; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_VERSION; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.EXTERNAL; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FILE; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FILE_STREAM; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FileInfo; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.GENERATION; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.INTERNAL; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.MASTER_URL; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.NO_INDEX_REPLICATION_REQUIRED; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.OFFSET; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.SIZE; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.TLOG_FILE; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.TLOG_FILES; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.getCheckSum; +import static org.apache.solr.common.params.CommonParams.JAVABIN; +import static org.apache.solr.common.params.CommonParams.NAME; + /** *

Provides functionality of downloading changed index files as well as config files and a timer for scheduling fetches from the * master.

@@ -188,6 +189,13 @@ class AlfrescoIndexFetcher private final Map confFileInfoCache = new HashMap<>(); private volatile Date replicationStartTimeStamp; private RTimer replicationTimer; + + /** + * The map contains the following fields: + * NAME : String -> file name(with path for contentstore files) + * SIZE : long -> file size + * CHECKSUM : long -> checksum + */ private volatile List> filesToDownload; private volatile List> confFilesToDownload; private volatile List> tlogFilesToDownload; @@ -197,6 +205,7 @@ class AlfrescoIndexFetcher private volatile List> confFilesDownloaded; private volatile List> tlogFilesDownloaded; private volatile List> contentStoreFilesDownloaded; + private volatile Map currentFile; private volatile DirectoryFileFetcher dirFileFetcher; private volatile LocalFsFileFetcher localFileFetcher; @@ -467,6 +476,8 @@ class AlfrescoIndexFetcher Map>> contentStoreMap = (Map>>) response .get(CONTENT_STORE_FILES); + fullContentStoreReplication = false; + if (contentStoreMap != null) { contentStoreFilesToDownload = Collections.synchronizedList(contentStoreMap.get(SolrContentStore.ADDS)); @@ -776,30 +787,37 @@ class AlfrescoIndexFetcher } } - if (contentStoreReplicationNeeded) - { + try { - if (contentStoreFilesToDownload != null) + if (contentStoreReplicationNeeded) { - bytesDownloaded += downloadContentStoreFiles(contentStore.getRootLocation()); + + if (contentStoreFilesToDownload != null) + { + bytesDownloaded += downloadContentStoreFiles(contentStore.getRootLocation()); + } + + if (contentStoreFilesToDelete != null) + { + deleteContentStoreFiles(contentStore.getRootLocation(), contentStoreFilesToDelete); + } + + if (fullContentStoreReplication) + { + cleanUpContentStore(contentStore.getRootLocation()); + } + + contentStore.setLastCommittedVersion(masterContentStoreVersion); + LOG.info("content store has been updated to version: {}", masterContentStoreVersion); } - if (contentStoreFilesToDelete != null) - { - deleteContentStoreFiles(contentStore.getRootLocation(), contentStoreFilesToDelete); - } - - if (fullContentStoreReplication) - { - cleanUpContentStore(contentStore.getRootLocation()); - } - - contentStore.setLastCommittedVersion(masterContentStoreVersion); + } catch (Exception e) { + LOG.error("impossible to complete content store replication {}", e); } final long timeTakenSeconds = getReplicationTimeElapsed(); final Long bytesDownloadedPerSecond = (timeTakenSeconds != 0 ? bytesDownloaded / timeTakenSeconds : - null); + null); LOG.info("Total time taken for download (fullCopy={},bytesDownloaded={}) : {} secs ({} bytes/sec)", isFullCopyNeeded, bytesDownloaded, timeTakenSeconds, bytesDownloadedPerSecond); @@ -1683,37 +1701,30 @@ class AlfrescoIndexFetcher * @param contentStorePath * @throws IOException */ - private void copyTmpContentStoreToContentStore(File tmpContentStoreDir, String contentStorePath) throws IOException + private void copyTmpContentStoreToContentStore(File tmpContentStoreDir, String contentStorePath) throws Exception { String tmpContentStorePath = tmpContentStoreDir.getPath(); - try - { - Files.walk(tmpContentStoreDir.toPath()).forEach(p -> { - File tmpFile = new File(p.toUri()); - if (!tmpFile.isDirectory()) + Files.walk(tmpContentStoreDir.toPath()).forEach(p -> { + File tmpFile = new File(p.toUri()); + if (!tmpFile.isDirectory()) + { + File csFile = new File(p.toString().replace(tmpContentStorePath, contentStorePath)); + try { - File csFile = new File(p.toString().replaceFirst(tmpContentStorePath, contentStorePath)); - try - { - Files.createDirectories(Paths.get(csFile.getParent())); - tmpFile.renameTo(csFile); - } - catch (IOException e) - { - LOG.error("impossible to copy {}", csFile.toString()); - } + Files.createDirectories(Paths.get(csFile.getParent())); + Files.move(tmpFile.toPath(), csFile.toPath(), StandardCopyOption.ATOMIC_MOVE); } - }); - } - catch (IOException e) - { - LOG.error("impossible tmp content store"); - throw e; - } + catch (IOException e) + { + throw new RuntimeException(e); + } + } + }); } + /** * Deletes the files in filesToDelete list from contentStore * @param contentStorePath @@ -1733,16 +1744,23 @@ class AlfrescoIndexFetcher * Deletes from contentstore all the files that has not been updated. * @param contentStorePath */ - private void cleanUpContentStore(String contentStorePath) + private void cleanUpContentStore(String contentStorePath) throws Exception { AtomicInteger fileDeleted = new AtomicInteger(); - Set fileNames = contentStoreFilesToDownload.stream().map(e -> (String) e.get(NAME)) + + // This is the set of the ONLY files that should be in contentStore. + // This set is computed from the information got from master. After a full replication, only the files + // that have been downloaded from master (contentStoreFilesToDownload) should be in contentStore. + // The file paths are translated in the current OS path notation. + Set contentStoreFiles = contentStoreFilesToDownload.stream() + .map(e -> (String) e.get(NAME)) + .map(FilenameUtils::separatorsToSystem) .collect(Collectors.toSet()); try { Files.walk(Paths.get(contentStorePath)).forEach(p -> { File f = new File(p.toUri()); - if (!f.isDirectory() && !fileNames.contains(p.toString().replaceFirst(contentStorePath, ""))) + if (!f.isDirectory() && !contentStoreFiles.contains(p.toString().replace(contentStorePath, ""))) { try { @@ -1756,9 +1774,10 @@ class AlfrescoIndexFetcher } }); } - catch (IOException e) + catch (Exception e) { LOG.error("Impossible to delete unnecessary files. Content store may contains unused contents"); + throw(e); } LOG.info("deleted {} unnecessary files from content store", fileDeleted); diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoReplicationHandler.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoReplicationHandler.java index be01b48b4..b382314e9 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoReplicationHandler.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoReplicationHandler.java @@ -2122,6 +2122,7 @@ public class AlfrescoReplicationHandler extends RequestHandlerBase implements So e.printStackTrace(); } } + } } catch (Exception e) @@ -2169,6 +2170,7 @@ public class AlfrescoReplicationHandler extends RequestHandlerBase implements So if (bytesRead <= 0) { writeNothingAndFlush(); + inputStream.close(); break; } @@ -2185,6 +2187,7 @@ public class AlfrescoReplicationHandler extends RequestHandlerBase implements So fos.flush(); } } + } } } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/lifecycle/SolrCoreLoadListener.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/lifecycle/SolrCoreLoadListener.java index 66ed18b9e..2a7c2b2ce 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/lifecycle/SolrCoreLoadListener.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/lifecycle/SolrCoreLoadListener.java @@ -21,6 +21,15 @@ package org.alfresco.solr.lifecycle; import static java.util.Arrays.asList; import static java.util.Optional.ofNullable; +import static org.alfresco.solr.SolrInformationServer.CASCADE_TRACKER_ENABLED; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.Properties; +import java.util.function.Function; +import java.util.function.Predicate; + import org.alfresco.opencmis.dictionary.CMISStrictDictionaryService; import org.alfresco.solr.AlfrescoCoreAdminHandler; import org.alfresco.solr.AlfrescoSolrDataModel; @@ -54,13 +63,6 @@ import org.apache.solr.search.SolrIndexSearcher; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import java.util.ArrayList; -import java.util.Collection; -import java.util.List; -import java.util.Properties; -import java.util.function.Function; -import java.util.function.Predicate; - /** * Listeners for *FIRST SEARCHER* events in order to prepare and register the SolrContentStore and the Tracking Subsystem. * @@ -251,25 +253,33 @@ public class SolrCoreLoadListener extends AbstractSolrEventListener MetadataTracker metadataTracker = registerAndSchedule( - new MetadataTracker(true, props, repositoryClient, core.getName(), srv), + new MetadataTracker(true, props, repositoryClient, core.getName(), srv, true), core, props, trackerRegistry, scheduler); - CascadeTracker cascadeTracker = - registerAndSchedule( - new CascadeTracker(props, repositoryClient, core.getName(), srv), - core, - props, - trackerRegistry, - scheduler); + List trackers = new ArrayList<>(); + + String cascadeTrackerEnabledProp = ofNullable((String) props.get(CASCADE_TRACKER_ENABLED)).orElse("true"); + if (Boolean.valueOf(cascadeTrackerEnabledProp)) + { + CascadeTracker cascadeTracker = + registerAndSchedule( + new CascadeTracker(props, repositoryClient, core.getName(), srv), + core, + props, + trackerRegistry, + scheduler); + trackers.add(cascadeTracker); + } //The CommitTracker will acquire these locks in order //The ContentTracker will likely have the longest runs so put it first to ensure the MetadataTracker is not paused while //waiting for the ContentTracker to release it's lock. //The aclTracker will likely have the shortest runs so put it last. - return asList(cascadeTracker, contentTracker, metadataTracker, aclTracker); + trackers.addAll(asList(contentTracker, metadataTracker, aclTracker)); + return trackers; } /** diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/AbstractTracker.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/AbstractTracker.java index d4429f92b..f4d381cd1 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/AbstractTracker.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/AbstractTracker.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2005-2014 Alfresco Software Limited. + * Copyright (C) 2005-2019 Alfresco Software Limited. * * This file is part of Alfresco * @@ -18,6 +18,8 @@ */ package org.alfresco.solr.tracker; +import static java.util.Optional.ofNullable; + import java.lang.invoke.MethodHandles; import java.net.ConnectException; import java.net.SocketTimeoutException; @@ -49,11 +51,10 @@ public abstract class AbstractTracker implements Tracker protected SOLRAPIClient client; InformationServer infoSrv; protected String coreName; - protected StoreRef storeRef; - protected long batchCount; - protected String alfrescoVersion; - protected TrackerStats trackerStats; - protected boolean runPostModelLoadInit = true; + StoreRef storeRef; + long batchCount; + TrackerStats trackerStats; + boolean runPostModelLoadInit = true; private int maxLiveSearchers; private volatile boolean shutdown = false; @@ -63,9 +64,9 @@ public abstract class AbstractTracker implements Tracker protected volatile TrackerState state; protected int shardCount; protected int shardInstance; - protected String shardMethod; + String shardMethod; protected boolean transformContent; - protected String shardTemplate; + String shardTemplate; protected volatile boolean rollback; protected final Type type; @@ -102,12 +103,8 @@ public abstract class AbstractTracker implements Tracker transformContent = Boolean.parseBoolean(p.getProperty("alfresco.index.transformContent", "true")); this.trackerStats = this.infoSrv.getTrackerStats(); - - alfrescoVersion = p.getProperty("alfresco.version", "5.0.0"); this.type = type; - - LOGGER.info("Solr built for Alfresco version: {}", alfrescoVersion); } @@ -183,12 +180,9 @@ public abstract class AbstractTracker implements Tracker if(this.state == null) { - /* - * Set the global state for the tracker here. - */ this.state = getTrackerState(); - LOGGER.debug("##### Setting tracker global state."); - LOGGER.debug("State set: {}", this.state.toString()); + + LOGGER.debug("Global Tracker State set to: {}", this.state.toString()); this.state.setRunning(true); } else @@ -237,12 +231,11 @@ public abstract class AbstractTracker implements Tracker finally { infoSrv.unregisterTrackerThread(); - if(state != null) - { - //During a rollback state is set to null. + ofNullable(state).ifPresent(tstate -> { + // During a rollback state is set to null. state.setRunning(false); state.setCheck(false); - } + }); runLock.release(); } } @@ -284,7 +277,7 @@ public abstract class AbstractTracker implements Tracker /** * Allows time for the scheduled asynchronous tasks to complete */ - protected synchronized void waitForAsynchronous() + synchronized void waitForAsynchronous() { AbstractWorkerRunnable currentRunnable = this.threadHandler.peekHeadReindexWorker(); while (currentRunnable != null) @@ -305,12 +298,12 @@ public abstract class AbstractTracker implements Tracker } } - public int getMaxLiveSearchers() + int getMaxLiveSearchers() { return maxLiveSearchers; } - protected void checkShutdown() + void checkShutdown() { if(shutdown) { @@ -345,20 +338,11 @@ public abstract class AbstractTracker implements Tracker return this.writeLock; } - public Semaphore getRunLock() + Semaphore getRunLock() { return this.runLock; } - /** - * @return Alfresco version Solr was built for - */ - @Override - public String getAlfrescoVersion() - { - return alfrescoVersion; - } - public Properties getProps() { return props; diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/CommitTracker.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/CommitTracker.java index 375c21ad8..6f7332035 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/CommitTracker.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/CommitTracker.java @@ -19,7 +19,11 @@ package org.alfresco.solr.tracker; +import static java.util.Optional.empty; +import static java.util.Optional.ofNullable; + import java.util.List; +import java.util.Optional; import java.util.Properties; import java.util.concurrent.atomic.AtomicInteger; @@ -41,7 +45,8 @@ public class CommitTracker extends AbstractTracker private MetadataTracker metadataTracker; private AclTracker aclTracker; private ContentTracker contentTracker; - private CascadeTracker cascadeTracker; + /** The cascade tracker. Note that this may be empty if cascade tracking is disabled. */ + private Optional cascadeTracker = empty(); private AtomicInteger rollbackCount = new AtomicInteger(0); protected final static Logger log = LoggerFactory.getLogger(CommitTracker.class); @@ -71,7 +76,7 @@ public class CommitTracker extends AbstractTracker } else if(tracker instanceof ContentTracker) { this.contentTracker = (ContentTracker)tracker; } else if(tracker instanceof CascadeTracker) { - this.cascadeTracker = (CascadeTracker)tracker; + this.cascadeTracker = ofNullable((CascadeTracker) tracker); } } @@ -182,8 +187,11 @@ public class CommitTracker extends AbstractTracker contentTracker.getWriteLock().acquire(); assert(contentTracker.getWriteLock().availablePermits() == 0); - cascadeTracker.getWriteLock().acquire(); - assert(cascadeTracker.getWriteLock().availablePermits() == 0); + if (cascadeTracker.isPresent()) + { + cascadeTracker.get().getWriteLock().acquire(); + assert (cascadeTracker.get().getWriteLock().availablePermits() == 0); + } infoSrv.rollback(); } @@ -206,12 +214,12 @@ public class CommitTracker extends AbstractTracker contentTracker.invalidateState(); //Reset cascadeTracker - cascadeTracker.setRollback(false); - cascadeTracker.invalidateState(); + cascadeTracker.ifPresent(c -> c.setRollback(false)); + cascadeTracker.ifPresent(c -> invalidateState()); //Release the locks contentTracker.getWriteLock().release(); - cascadeTracker.getWriteLock().release(); + cascadeTracker.ifPresent(c -> c.getWriteLock().release()); rollbackCount.incrementAndGet(); } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/MetadataTracker.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/MetadataTracker.java index 3862ebd30..3a7773dd9 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/MetadataTracker.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/MetadataTracker.java @@ -32,6 +32,7 @@ import org.alfresco.repo.index.shard.ShardState; import org.alfresco.solr.BoundedDeque; import org.alfresco.solr.InformationServer; import org.alfresco.solr.NodeReport; +import org.alfresco.solr.SolrInformationServer; import org.alfresco.solr.TrackerState; import org.alfresco.solr.adapters.IOpenBitSet; import org.alfresco.solr.client.GetNodesParameters; @@ -40,6 +41,7 @@ import org.alfresco.solr.client.Node.SolrApiNodeStatus; import org.alfresco.solr.client.SOLRAPIClient; import org.alfresco.solr.client.Transaction; import org.alfresco.solr.client.Transactions; +import org.alfresco.util.Pair; import org.apache.commons.codec.EncoderException; import org.json.JSONException; import org.slf4j.Logger; @@ -63,14 +65,89 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker private ConcurrentLinkedQueue nodesToIndex = new ConcurrentLinkedQueue<>(); private ConcurrentLinkedQueue nodesToPurge = new ConcurrentLinkedQueue<>(); private ConcurrentLinkedQueue queriesToReindex = new ConcurrentLinkedQueue<>(); + + /** + * Check if nextTxCommitTimeService is available in the repository. + * This service is used to find the next available transaction commit time from a given time, + * so periods of time where no document updating is happening can be skipped while getting + * pending transactions list. + * + * {@link org.alfresco.solr.client.SOLRAPIClient#GET_NEXT_TX_COMMIT_TIME} + */ + private boolean nextTxCommitTimeServiceAvailable = false; + + /** + * Check if txInteravlCommitTimeService is available in the repository. + * This service returns the minimum and the maximum commit time for transactions in a node id range, + * so method sharding DB_ID_RANGE can skip transactions not relevant for the DB ID range. + * + * {@link org.alfresco.solr.client.SOLRAPIClient#GET_TX_INTERVAL_COMMIT_TIME} + */ + private boolean txIntervalCommitTimeServiceAvailable = false; + /** Whether the cascade tracking is enabled. */ + private boolean cascadeTrackerEnabled = true; public MetadataTracker(final boolean isMaster, Properties p, SOLRAPIClient client, String coreName, - InformationServer informationServer) + InformationServer informationServer) + { + this(isMaster, p, client, coreName, informationServer, false); + } + + /** + * MetadataTracker constructor + * + * @param isMaster is true if SOLR instance is master, false otherwise + * @param p includes SOLR core properties (from environment variables and properties file) + * @param client Alfresco Repository http client + * @param coreName Name of the SOLR Core (alfresco, archive) + * @param informationServer SOLR Information Server + * @param checkRepoServicesAvailability is true if Repo Services availability needs to be checked + */ + public MetadataTracker(final boolean isMaster, Properties p, SOLRAPIClient client, String coreName, + InformationServer informationServer, boolean checkRepoServicesAvailability) { super(isMaster, p, client, coreName, informationServer, Tracker.Type.METADATA); transactionDocsBatchSize = Integer.parseInt(p.getProperty("alfresco.transactionDocsBatchSize", "100")); nodeBatchSize = Integer.parseInt(p.getProperty("alfresco.nodeBatchSize", "10")); threadHandler = new ThreadHandler(p, coreName, "MetadataTracker"); + cascadeTrackerEnabled = informationServer.cascadeTrackingEnabled(); + + // In order to apply performance optimizations, checking the availability of Repo Web Scripts is required. + // As these services are available from ACS 6.2 + if (checkRepoServicesAvailability) + { + // Try invoking getNextTxCommitTime service + try + { + client.getNextTxCommitTime(coreName, 0l); + nextTxCommitTimeServiceAvailable = true; + } + catch (NoSuchMethodException e) + { + log.warn("nextTxCommitTimeService is not available. Upgrade your ACS Repository version in order to use this feature: {} ", e.getMessage()); + } + catch (Exception e) + { + log.error("Checking nextTxCommitTimeService failed.", e); + } + + // Try invoking txIntervalCommitTime service + try + { + client.getTxIntervalCommitTime(coreName, 0l, 0l); + txIntervalCommitTimeServiceAvailable = true; + } + catch (NoSuchMethodException e) + { + log.warn("txIntervalCommitTimeServiceAvailable is not available. If you are using DB_ID_RANGE shard method, " + + "upgrade your ACS Repository version in order to use this feature: {} ", e.getMessage()); + } + catch (Exception e) + { + log.error("Checking txIntervalCommitTimeServiceAvailable failed.", e); + } + } + } MetadataTracker() @@ -518,7 +595,7 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker } protected Transactions getSomeTransactions(BoundedDeque txnsFound, Long fromCommitTime, long timeStep, - int maxResults, long endTime) throws AuthenticationException, IOException, JSONException, EncoderException + int maxResults, long endTime) throws AuthenticationException, IOException, JSONException, EncoderException, NoSuchMethodException { long actualTimeStep = timeStep; @@ -546,6 +623,17 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker { transactions = client.getTransactions(startTime, null, startTime + actualTimeStep, null, maxResults, shardstate); startTime += actualTimeStep; + + // If no transactions are found, advance the time window to the next available transaction commit time + if (nextTxCommitTimeServiceAvailable && transactions.getTransactions().size() == 0) + { + Long nextTxCommitTime = client.getNextTxCommitTime(coreName, startTime); + if (nextTxCommitTime != -1) + { + log.info("Advancing transactions from {} to {}", startTime, nextTxCommitTime); + transactions = client.getTransactions(nextTxCommitTime, null, nextTxCommitTime + actualTimeStep, null, maxResults, shardstate); + } + } } while (((transactions.getTransactions().size() == 0) && (startTime < endTime)) || ((transactions.getTransactions().size() > 0) && alreadyFoundTransactions(txnsFound, transactions))); @@ -605,9 +693,46 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker * */ - Long fromCommitTime = getTxFromCommitTime(txnsFound, state.getLastGoodTxCommitTimeInIndex()); + Long fromCommitTime = getTxFromCommitTime(txnsFound, + state.getLastIndexedTxCommitTime() == 0 ? state.getLastGoodTxCommitTimeInIndex() : state.getLastIndexedTxCommitTime()); log.debug("#### Check txnsFound : " + txnsFound.size()); log.debug("======= fromCommitTime: " + fromCommitTime); + + // When using DB_ID_RANGE, fromCommitTime cannot be before the commit time of the first transaction + // for the DB_ID_RANGE to be indexed and commit time of the last transaction cannot be lower than fromCommitTime. + // When there isn't nodes in that range, -1 is returned as commit times + if (docRouter instanceof DBIDRangeRouter && txIntervalCommitTimeServiceAvailable) + { + + DBIDRangeRouter dbIdRangeRouter = (DBIDRangeRouter) docRouter; + Pair commitTimes = client.getTxIntervalCommitTime(coreName, + dbIdRangeRouter.getStartRange(), dbIdRangeRouter.getEndRange()); + Long shardMinCommitTime = commitTimes.getFirst(); + Long shardMaxCommitTime = commitTimes.getSecond(); + + // Node Range it's not still available in repository + if (shardMinCommitTime == -1) + { + log.debug("#### [DB_ID_RANGE] No nodes in range [" + dbIdRangeRouter.getStartRange() + "-" + + dbIdRangeRouter.getEndRange() + "] " + + "exist in the repository. Skipping metadata tracking."); + return; + } + if (fromCommitTime > shardMaxCommitTime) + { + log.debug("#### [DB_ID_RANGE] Last commit time is greater that max commit time in in range [" + + dbIdRangeRouter.getStartRange() + "-" + dbIdRangeRouter.getEndRange() + "]. " + + "Skipping metadata tracking."); + return; + } + // Initial commit time for Node Range is greater than calculated from commit time + if (fromCommitTime < shardMinCommitTime) + { + log.debug("#### [DB_ID_RANGE] SKIPPING TRANSACTIONS FROM " + fromCommitTime + " TO " + + shardMinCommitTime); + fromCommitTime = shardMinCommitTime; + } + } log.debug("#### Get txn from commit time: " + fromCommitTime); transactions = getSomeTransactions(txnsFound, fromCommitTime, TIME_STEP_1_HR_IN_MS, 2000, @@ -836,7 +961,7 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker List filteredNodes = filterNodes(nodes); if(filteredNodes.size() > 0) { - this.infoServer.indexNodes(filteredNodes, true, false); + this.infoServer.indexNodes(filteredNodes, true); } } @@ -856,9 +981,8 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker { filteredList.add(node); } - else + else if (cascadeTrackerEnabled) { - if(node.getStatus() == SolrApiNodeStatus.UPDATED) { Node doCascade = new Node(); @@ -964,7 +1088,7 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker } public IndexHealthReport checkIndex(Long toTx, Long toAclTx, Long fromTime, Long toTime) - throws IOException, AuthenticationException, JSONException, EncoderException + throws IOException, AuthenticationException, JSONException, EncoderException, NoSuchMethodException { // DB TX Count long firstTransactionCommitTime = 0; diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/SlaveCoreStatePublisher.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/SlaveCoreStatePublisher.java index 19a93631c..71750c53f 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/SlaveCoreStatePublisher.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/SlaveCoreStatePublisher.java @@ -1,3 +1,21 @@ +/* + * Copyright (C) 2005-2019 Alfresco Software Limited. + * + * This file is part of Alfresco + * + * Alfresco is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Alfresco is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with Alfresco. If not, see . + */ package org.alfresco.solr.tracker; import static org.alfresco.solr.tracker.Tracker.Type.NODE_STATE_PUBLISHER; diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/Tracker.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/Tracker.java index 05fe0766a..976d52c20 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/Tracker.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/Tracker.java @@ -32,8 +32,6 @@ public interface Tracker boolean hasMaintenance() throws Exception; Semaphore getWriteLock(); - - String getAlfrescoVersion(); void setShutdown(boolean shutdown); diff --git a/search-services/alfresco-search/src/main/resources/solr/instance/conf/shared.properties b/search-services/alfresco-search/src/main/resources/solr/instance/conf/shared.properties index e78b8ee32..63bc6bfe2 100644 --- a/search-services/alfresco-search/src/main/resources/solr/instance/conf/shared.properties +++ b/search-services/alfresco-search/src/main/resources/solr/instance/conf/shared.properties @@ -30,4 +30,7 @@ alfresco.cross.locale.property.1={http://www.alfresco.org/model/content/1.0}lock # alfresco.cross.locale.datatype.1={http://www.alfresco.org/model/dictionary/1.0}content # alfresco.cross.locale.datatype.2={http://www.alfresco.org/model/dictionary/1.0}mltext -alfresco.model.tracker.cron=0/10 * * * * ? * \ No newline at end of file +alfresco.model.tracker.cron=0/10 * * * * ? * + +# Whether path queries are enabled. +alfresco.cascade.tracker.enabled=true diff --git a/search-services/alfresco-search/src/main/resources/solr/instance/templates/noRerank/conf/solrconfig.xml b/search-services/alfresco-search/src/main/resources/solr/instance/templates/noRerank/conf/solrconfig.xml index a51492e56..e7bd84f6c 100644 --- a/search-services/alfresco-search/src/main/resources/solr/instance/templates/noRerank/conf/solrconfig.xml +++ b/search-services/alfresco-search/src/main/resources/solr/instance/templates/noRerank/conf/solrconfig.xml @@ -1154,7 +1154,7 @@ https://wiki.apache.org/solr/SolrCloud/ --> - + + 1.5.0 unpack-solr-war @@ -111,7 +111,8 @@ ${project.version} libs ${project.build.directory}/solr-libs - **/jackson-dataformat-smile-*.jar,**/asm-3.3.1.jar,**/jackson-core-asl-*.jar,**/jackson-mapper-asl-*.jar,**/dom4j-1.6.1.jar,**/annotations-1.0.0.jar + **/jackson-dataformat-smile-*.jar,**/asm-3.3.1.jar,**/jackson-core-asl-*.jar,**/jackson-mapper-asl-*.jar,**/dom4j-1.6.1.jar, + **/annotations-1.0.0.jar,**/spring-context-support-*.jar,**/spring-web-*.jar,**/woodstox-core-asl-4.4.1.jar @@ -157,7 +158,9 @@ + + diff --git a/search-services/packaging/src/docker/6.x/docker-compose.yml b/search-services/packaging/src/docker/6.x/docker-compose.yml index 1174827d3..ca4268e79 100644 --- a/search-services/packaging/src/docker/6.x/docker-compose.yml +++ b/search-services/packaging/src/docker/6.x/docker-compose.yml @@ -37,6 +37,10 @@ services: search: image: quay.io/alfresco/search-services:${SEARCH_TAG} environment: + #Replication properties + - REPLICATION_TYPE=master + #- REPLICATION_AFTER=commit,startup- SOLR_ALFRESCO_HOST=alfresco + #- REPLICATION_CONFIG_FILES=schema.xml,stopwords.txt- SOLR_ALFRESCO_PORT=8080 #Solr needs to know how to register itself with Alfresco - SOLR_ALFRESCO_HOST=alfresco - SOLR_ALFRESCO_PORT=8080 @@ -51,6 +55,27 @@ services: - ENABLE_SPELLCHECK=${SEARCH_ENABLE_SPELLCHECK} ports: - 8083:8983 #Browser port + #search_slave: + # image: quay.io/alfresco/search-services:${SEARCH_TAG} + # environment: + # #Replication properties + # - REPLICATION_TYPE=slave + # - REPLICATION_MASTER_HOST=search + # - REPLICATION_MASTER_PORT=8983 + # #- REPLICATION_MASTER_PROTOCOL=http + # #- REPLICATION_CORE_NAME=alfresco + # #- REPLICATION_POLL_INTERVAL=00:00:60 + # #Solr needs to know how to register itself with Alfresco + # - SOLR_ALFRESCO_HOST=alfresco + # - SOLR_ALFRESCO_PORT=8080 + # #Alfresco needs to know how to call solr + # - SOLR_SOLR_HOST=search + # - SOLR_SOLR_PORT=8983 + # #Create the default alfresco and archive cores + # - SOLR_CREATE_ALFRESCO_DEFAULTS=alfresco,archive + # ports: + # - 8084:8983 #Browser port + activemq: image: alfresco/alfresco-activemq:5.15.6 ports: diff --git a/search-services/packaging/src/docker/Dockerfile b/search-services/packaging/src/docker/Dockerfile index 852223ecf..ea60bca76 100644 --- a/search-services/packaging/src/docker/Dockerfile +++ b/search-services/packaging/src/docker/Dockerfile @@ -1,6 +1,6 @@ # Alfresco Search Services ${project.version} Docker Image -FROM alfresco/alfresco-base-java:11.0.1-openjdk-centos-7-6784d76a7b81 +FROM alfresco/alfresco-base-java:11.0.1-openjdk-centos-7-7a6031154417 LABEL creator="Gethin James" maintainer="Alfresco Search Services Team" ENV DIST_DIR /opt/alfresco-search-services diff --git a/search-services/packaging/src/docker/search_config_setup.sh b/search-services/packaging/src/docker/search_config_setup.sh index 40ad069e1..12fc0bf36 100644 --- a/search-services/packaging/src/docker/search_config_setup.sh +++ b/search-services/packaging/src/docker/search_config_setup.sh @@ -1,5 +1,69 @@ #!/bin/bash set -e +# By default its going to deploy "Master" setup configuration with "REPLICATION_TYPE=master". +# Slave replica service can be enabled using "REPLICATION_TYPE=slave" environment value. + +RERANK_TEMPLATE_PATH=$PWD/solrhome/templates/rerank/conf +NORERANK_TEMPLATE_PATH=$PWD/solrhome/templates/noRerank/conf +SOLR_RERANK_CONFIG_FILE=$RERANK_TEMPLATE_PATH/solrconfig.xml +SOLR_NORERANK_CONFIG_FILE=$NORERANK_TEMPLATE_PATH/solrconfig.xml +SOLR_RERANK_CORE_FILE=$RERANK_TEMPLATE_PATH/solrcore.properties +SOLR_NORERANK_CORE_FILE=$NORERANK_TEMPLATE_PATH/solrcore.properties + +if [[ $REPLICATION_TYPE == "master" ]]; then + + findStringMaster='' + + replaceStringMaster="\n\t \n" + + if [[ $REPLICATION_AFTER == "" ]]; then + REPLICATION_AFTER=commit,startup + fi + + if [[ $REPLICATION_CONFIG_FILES == "" ]]; then + REPLICATION_CONFIG_FILES=schema.xml,stopwords.txt + fi + + for i in $(echo $REPLICATION_AFTER | sed "s/,/ /g") + do + replaceStringMaster+="\t\t"$i"<\/str> \n" + done + + if [[ ! -z "$REPLICATION_CONFIG_FILES" ]]; then + replaceStringMaster+="\t\t$REPLICATION_CONFIG_FILES<\/str> \n" + fi + + replaceStringMaster+="\t<\/lst>" + + sed -i "s/$findStringMaster/$findStringMaster$replaceStringMaster/g" $SOLR_RERANK_CONFIG_FILE $SOLR_NORERANK_CONFIG_FILE + sed -i "s/enable.alfresco.tracking=true/enable.alfresco.tracking=true\nenable.master=true\nenable.slave=false/g" $SOLR_RERANK_CORE_FILE $SOLR_NORERANK_CORE_FILE +fi + +if [[ $REPLICATION_TYPE == "slave" ]]; then + + if [[ $REPLICATION_MASTER_PROTOCOL != https ]]; then + REPLICATION_MASTER_PROTOCOL=http + fi + + if [[ $REPLICATION_MASTER_HOST == "" ]]; then + REPLICATION_MASTER_HOST=localhost + fi + + if [[ $REPLICATION_MASTER_PORT == "" ]]; then + REPLICATION_MASTER_PORT=8083 + fi + + if [[ $REPLICATION_POLL_INTERVAL == "" ]]; then + REPLICATION_POLL_INTERVAL=00:00:30 + fi + + sed -i 's//\ + \ + '$REPLICATION_MASTER_PROTOCOL':\/\/'$REPLICATION_MASTER_HOST':'$REPLICATION_MASTER_PORT'\/solr\/${solr.core.name}<\/str>\ + '$REPLICATION_POLL_INTERVAL'<\/str>\ + <\/lst>/g' $SOLR_RERANK_CONFIG_FILE $SOLR_NORERANK_CONFIG_FILE + sed -i "s/enable.alfresco.tracking=true/enable.alfresco.tracking=false\nenable.master=false\nenable.slave=true/g" $SOLR_RERANK_CORE_FILE $SOLR_NORERANK_CORE_FILE +fi SOLR_IN_FILE=$PWD/solr.in.sh @@ -21,8 +85,7 @@ fi # By default Docker Image is using TLS Mutual Authentication (SSL) for communications with Repository # Plain HTTP can be enabled by setting ALFRESCO_SECURE_COMMS to 'none' if [[ "none" == "$ALFRESCO_SECURE_COMMS" ]]; then - sed -i 's/alfresco.secureComms=https/alfresco.secureComms=none/' ${PWD}/solrhome/templates/rerank/conf/solrcore.properties - sed -i 's/alfresco.secureComms=https/alfresco.secureComms=none/' ${PWD}/solrhome/templates/noRerank/conf/solrcore.properties + sed -i 's/alfresco.secureComms=https/alfresco.secureComms=none/' $SOLR_RERANK_CORE_FILE $SOLR_NORERANK_CORE_FILE # Apply also the setting to existing SOLR cores property files when existing if [[ -f ${PWD}/solrhome/alfresco/conf/solrcore.properties ]]; then sed -i 's/alfresco.secureComms=https/alfresco.secureComms=none/' ${PWD}/solrhome/alfresco/conf/solrcore.properties diff --git a/search-services/packaging/src/main/resources/licenses/3rd-party/Apache-like-XPP.txt b/search-services/packaging/src/main/resources/licenses/3rd-party/BSDlike-XPP.txt similarity index 89% rename from search-services/packaging/src/main/resources/licenses/3rd-party/Apache-like-XPP.txt rename to search-services/packaging/src/main/resources/licenses/3rd-party/BSDlike-XPP.txt index fb11965fa..e4553dcaf 100644 --- a/search-services/packaging/src/main/resources/licenses/3rd-party/Apache-like-XPP.txt +++ b/search-services/packaging/src/main/resources/licenses/3rd-party/BSDlike-XPP.txt @@ -1,58 +1,58 @@ -LICENSE FOR THE Extreme! Lab PullParser ------------------------------------------------------------------------- - -Copyright © 2002 The Trustees of Indiana University. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - -1) All redistributions of source code must retain the above - copyright notice, the list of authors in the original source - code, this list of conditions and the disclaimer listed in this - license; - -2) All redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the disclaimer - listed in this license in the documentation and/or other - materials provided with the distribution; - -3) Any documentation included with all redistributions must include - the following acknowledgement: - - "This product includes software developed by the Indiana - University Extreme! Lab. For further information please visit - http://www.extreme.indiana.edu/" - - Alternatively, this acknowledgment may appear in the software - itself, and wherever such third-party acknowledgments normally - appear. - -4) The name "Indiana Univeristy" and "Indiana Univeristy - Extreme! Lab" shall not be used to endorse or promote - products derived from this software without prior written - permission from Indiana University. For written permission, - please contact http://www.extreme.indiana.edu/. - -5) Products derived from this software may not use "Indiana - Univeristy" name nor may "Indiana Univeristy" appear in their name, - without prior written permission of the Indiana University. - -Indiana University provides no reassurances that the source code -provided does not infringe the patent or any other intellectual -property rights of any other entity. Indiana University disclaims any -liability to any recipient for claims brought by any other entity -based on infringement of intellectual property rights or otherwise. - -LICENSEE UNDERSTANDS THAT SOFTWARE IS PROVIDED "AS IS" FOR WHICH -NO WARRANTIES AS TO CAPABILITIES OR ACCURACY ARE MADE. INDIANA -UNIVERSITY GIVES NO WARRANTIES AND MAKES NO REPRESENTATION THAT -SOFTWARE IS FREE OF INFRINGEMENT OF THIRD PARTY PATENT, COPYRIGHT, OR -OTHER PROPRIETARY RIGHTS.  INDIANA UNIVERSITY MAKES NO WARRANTIES THAT -SOFTWARE IS FREE FROM "BUGS", "VIRUSES", "TROJAN HORSES", "TRAP -DOORS", "WORMS", OR OTHER HARMFUL CODE. LICENSEE ASSUMES THE ENTIRE -RISK AS TO THE PERFORMANCE OF SOFTWARE AND/OR ASSOCIATED MATERIALS, -AND TO THE PERFORMANCE AND VALIDITY OF INFORMATION GENERATED USING -SOFTWARE. - +LICENSE FOR THE Extreme! Lab +------------------------------------------------------------------------ + +Copyright © 2003 The Trustees of Indiana University. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1) All redistributions of source code must retain the above + copyright notice, the list of authors in the original source + code, this list of conditions and the disclaimer listed in this + license; + +2) All redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the disclaimer + listed in this license in the documentation and/or other + materials provided with the distribution; + +3) Any documentation included with all redistributions must include + the following acknowledgement: + + "This product includes software developed by the Indiana + University Extreme! Lab. For further information please visit + http://www.extreme.indiana.edu/" + + Alternatively, this acknowledgment may appear in the software + itself, and wherever such third-party acknowledgments normally + appear. + +4) The name "Indiana University" and "Indiana University + Extreme! Lab" shall not be used to endorse or promote + products derived from this software without prior written + permission from Indiana University. For written permission, + please contact http://www.extreme.indiana.edu/. + +5) Products derived from this software may not use "Indiana + University" name nor may "Indiana University" appear in their name, + without prior written permission of the Indiana University. + +Indiana University provides no reassurances that the source code +provided does not infringe the patent or any other intellectual +property rights of any other entity. Indiana University disclaims any +liability to any recipient for claims brought by any other entity +based on infringement of intellectual property rights or otherwise. + +LICENSEE UNDERSTANDS THAT SOFTWARE IS PROVIDED "AS IS" FOR WHICH +NO WARRANTIES AS TO CAPABILITIES OR ACCURACY ARE MADE. INDIANA +UNIVERSITY GIVES NO WARRANTIES AND MAKES NO REPRESENTATION THAT +SOFTWARE IS FREE OF INFRINGEMENT OF THIRD PARTY PATENT, COPYRIGHT, OR +OTHER PROPRIETARY RIGHTS.  INDIANA UNIVERSITY MAKES NO WARRANTIES THAT +SOFTWARE IS FREE FROM "BUGS", "VIRUSES", "TROJAN HORSES", "TRAP +DOORS", "WORMS", OR OTHER HARMFUL CODE. LICENSEE ASSUMES THE ENTIRE +RISK AS TO THE PERFORMANCE OF SOFTWARE AND/OR ASSOCIATED MATERIALS, +AND TO THE PERFORMANCE AND VALIDITY OF INFORMATION GENERATED USING +SOFTWARE. + diff --git a/search-services/packaging/src/main/resources/licenses/notice.txt b/search-services/packaging/src/main/resources/licenses/notice.txt index 92c61b013..b3eb46870 100644 --- a/search-services/packaging/src/main/resources/licenses/notice.txt +++ b/search-services/packaging/src/main/resources/licenses/notice.txt @@ -22,8 +22,8 @@ antlr-3.5.2.jar http://www.antlr.org/ jaxen-1.2.0.jar http://www.cafeconleche.org/jaxen/ -=== Apache variant License === -xpp3-1.1.3_8.jar http://www.extreme.indiana.edu/xgws/xsoap/xpp/ +=== BSD variant License === +xpp3-1.1.4c.jar http://www.extreme.indiana.edu/dist/java-repository/xpp3/licenses/LICENSE.txt === JSON === @@ -32,43 +32,38 @@ json-20160212.jar http://code.google.com/p/json-simple/ === Apache 2.0 === xml-resolver-1.2.jar https://github.com/FasterXML/jackson -neethi-3.0.3.jar http://ws.apache.org/commons/neethi/ +neethi-3.1.1.jar http://ws.apache.org/commons/neethi/ commons-logging-1.2.jar http://jakarta.apache.org/commons/ commons-lang3-3.9.jar http://jakarta.apache.org/commons/ -mybatis-3.3.0.jar http://www.mybatis.org/ chemistry-opencmis-commons-impl-1.1.0.jar http://chemistry.apache.org/ chemistry-opencmis-commons-api-1.1.0.jar http://chemistry.apache.org/ -xmlschema-core-2.2.1.jar http://ws.apache.org/commons/XmlSchema/ +xmlschema-core-2.2.5.jar http://ws.apache.org/commons/XmlSchema/ HikariCP-java7-2.4.13.jar https://github.com/brettwooldridge/HikariCP -cxf-core-3.0.12.jar https://cxf.apache.org/ -cxf-rt-bindings-soap-3.0.12.jar https://cxf.apache.org/ -cxf-rt-bindings-xml-3.0.12.jar https://cxf.apache.org/ -cxf-rt-databinding-jaxb-3.0.12.jar https://cxf.apache.org/ -cxf-rt-frontend-jaxws-3.0.12.jar https://cxf.apache.org/ -cxf-rt-frontend-simple-3.0.12.jar https://cxf.apache.org/ -cxf-rt-transports-http-3.0.12.jar https://cxf.apache.org/ -cxf-rt-ws-addr-3.0.12.jar https://cxf.apache.org/ -cxf-rt-ws-policy-3.0.12.jar https://cxf.apache.org/ -cxf-rt-wsdl-3.0.12.jar https://cxf.apache.org/ -mybatis-spring-1.2.5.jar http://www.mybatis.org/ +cxf-core-3.2.12.jar https://cxf.apache.org/ +cxf-rt-bindings-soap-3.2.12.jar https://cxf.apache.org/ +cxf-rt-bindings-xml-3.2.12.jar https://cxf.apache.org/ +cxf-rt-databinding-jaxb-3.2.12.jar https://cxf.apache.org/ +cxf-rt-frontend-jaxws-3.2.12.jar https://cxf.apache.org/ +cxf-rt-frontend-simple-3.2.12.jar https://cxf.apache.org/ +cxf-rt-transports-http-3.2.12.jar https://cxf.apache.org/ +cxf-rt-ws-addr-3.2.12.jar https://cxf.apache.org/ +cxf-rt-ws-policy-3.2.12.jar https://cxf.apache.org/ +cxf-rt-wsdl-3.2.12.jar https://cxf.apache.org/ chemistry-opencmis-server-support-1.0.0.jar http://chemistry.apache.org/ chemistry-opencmis-server-bindings-1.0.0.jar http://chemistry.apache.org/ -quartz-2.3.1.jar http://quartz-scheduler.org/ -jackson-core-2.10.1.jar https://github.com/FasterXML/jackson -jackson-annotations-2.10.1.jar https://github.com/FasterXML/jackson -jackson-databind-2.10.1.jar https://github.com/FasterXML/jackson +quartz-2.3.2.jar http://quartz-scheduler.org/ +jackson-core-2.10.2.jar https://github.com/FasterXML/jackson +jackson-annotations-2.10.2.jar https://github.com/FasterXML/jackson +jackson-databind-2.10.2.jar https://github.com/FasterXML/jackson commons-httpclient-3.1-HTTPCLIENT-1265.jar http://jakarta.apache.org/commons/ -spring-aop-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-beans-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-context-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-context-support-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-core-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-expression-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-jcl-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-jdbc-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-orm-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-tx-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-web-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-aop-5.2.3.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-beans-5.2.3.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-context-5.2.3.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-core-5.2.3.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-expression-5.2.3.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-jdbc-5.2.3.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-orm-5.2.3.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-tx-5.2.3.RELEASE.jar http://projects.spring.io/spring-framework/ xercesImpl-2.12.0-alfresco-patched-20191004.jar http://xerces.apache.org/xerces2-j guessencoding-1.4.jar http://docs.codehaus.org/display/GUESSENC/ xml-apis-1.4.01.jar https://github.com/FasterXML/jackson @@ -87,6 +82,7 @@ jetty-servlets-9.3.27.v20190418.jar https://www.eclipse.org/jetty/licenses.html jetty-util-9.3.27.v20190418.jar https://www.eclipse.org/jetty/licenses.html jetty-webapp-9.3.27.v20190418.jar https://www.eclipse.org/jetty/licenses.html jetty-xml-9.3.27.v20190418.jar https://www.eclipse.org/jetty/licenses.html +woodstox-core-5.0.3.jar https://github.com/FasterXML/woodstox === CDDL 1.0 === @@ -115,8 +111,8 @@ asm-commons-5.1.jar aspectjrt-1.8.0.jar attributes-binder-1.3.1.jar avatica-core-1.9.0.jar -bcmail-jdk15-1.45.jar -bcprov-jdk15-1.45.jar +bcmail-jdk15on-1.47.jar +bcprov-jdk15on-1.47.jar boilerpipe-1.1.0.jar caffeine-2.4.0.jar calcite-core-1.11.0.jar @@ -131,7 +127,7 @@ commons-compiler-2.7.6.jar commons-compress-1.14.jar commons-configuration-1.6.jar commons-exec-1.3.jar -commons-fileupload-1.3.2.jar +commons-fileupload-1.3.3.jar commons-io-2.5.jar commons-lang-2.6.jar commons-math3-3.4.1.jar @@ -168,28 +164,28 @@ jul-to-slf4j-1.7.7.jar juniversalchardet-1.0.3.jar langdetect-1.1-20120112.jar log4j-1.2.17.jar -lucene-analyzers-common-6.6.5-patched.1.jar -lucene-analyzers-icu-6.6.5-patched.1.jar -lucene-analyzers-kuromoji-6.6.5-patched.1.jar -lucene-analyzers-morfologik-6.6.5-patched.1.jar -lucene-analyzers-phonetic-6.6.5-patched.1.jar -lucene-analyzers-smartcn-6.6.5-patched.1.jar -lucene-analyzers-stempel-6.6.5-patched.1.jar -lucene-backward-codecs-6.6.5-patched.1.jar -lucene-classification-6.6.5-patched.1.jar -lucene-codecs-6.6.5-patched.1.jar -lucene-core-6.6.5-patched.1.jar -lucene-expressions-6.6.5-patched.1.jar -lucene-grouping-6.6.5-patched.1.jar -lucene-highlighter-6.6.5-patched.1.jar -lucene-join-6.6.5-patched.1.jar -lucene-memory-6.6.5-patched.1.jar -lucene-misc-6.6.5-patched.1.jar -lucene-queries-6.6.5-patched.1.jar -lucene-queryparser-6.6.5-patched.1.jar -lucene-sandbox-6.6.5-patched.1.jar -lucene-spatial-extras-6.6.5-patched.1.jar -lucene-suggest-6.6.5-patched.1.jar +lucene-analyzers-common-6.6.5-patched.2.jar +lucene-analyzers-icu-6.6.5-patched.2.jar +lucene-analyzers-kuromoji-6.6.5-patched.2.jar +lucene-analyzers-morfologik-6.6.5-patched.2.jar +lucene-analyzers-phonetic-6.6.5-patched.2.jar +lucene-analyzers-smartcn-6.6.5-patched.2.jar +lucene-analyzers-stempel-6.6.5-patched.2.jar +lucene-backward-codecs-6.6.5-patched.2.jar +lucene-classification-6.6.5-patched.2.jar +lucene-codecs-6.6.5-patched.2.jar +lucene-core-6.6.5-patched.2.jar +lucene-expressions-6.6.5-patched.2.jar +lucene-grouping-6.6.5-patched.2.jar +lucene-highlighter-6.6.5-patched.2.jar +lucene-join-6.6.5-patched.2.jar +lucene-memory-6.6.5-patched.2.jar +lucene-misc-6.6.5-patched.2.jar +lucene-queries-6.6.5-patched.2.jar +lucene-queryparser-6.6.5-patched.2.jar +lucene-sandbox-6.6.5-patched.2.jar +lucene-spatial-extras-6.6.5-patched.2.jar +lucene-suggest-6.6.5-patched.2.jar metadata-extractor-2.9.1.jar metrics-core-3.2.2.jar metrics-ganglia-3.2.2.jar @@ -213,11 +209,11 @@ rome-1.5.1.jar simple-xml-2.7.1.jar slf4j-api-1.7.7.jar slf4j-log4j12-1.7.7.jar -solr-analysis-extras-6.6.5-patched.1.jar -solr-clustering-6.6.5-patched.1.jar -solr-core-6.6.5-patched.1.jar -solr-langid-6.6.5-patched.1.jar -solr-solrj-6.6.5-patched.1.jar +solr-analysis-extras-6.6.5-patched.2.jar +solr-clustering-6.6.5-patched.2.jar +solr-core-6.6.5-patched.2.jar +solr-langid-6.6.5-patched.2.jar +solr-solrj-6.6.5-patched.2.jar spatial4j-0.6.jar start.jar stax2-api-3.1.4.jar @@ -229,7 +225,6 @@ tika-parsers-1.16.jar tika-xmp-1.16.jar vorbis-java-core-0.8.jar vorbis-java-tika-0.8.jar -woodstox-core-asl-4.4.1.jar xmlbeans-2.6.0.jar xmpcore-5.1.2.jar xz-1.6.jar diff --git a/search-services/pom.xml b/search-services/pom.xml index 8af1e3219..53e3c12de 100644 --- a/search-services/pom.xml +++ b/search-services/pom.xml @@ -14,7 +14,8 @@ pom Alfresco Solr Search parent - 1.7.29 + 1.7.30 + 3.2.12 alfresco-solrclient-lib