diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..575155bab --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,67 @@ +# Contributing + +Thanks for your interest in contributing to this project! + +The following is a set of guidelines for contributing to this library. Most of them will +make the life of the reviewer easier and therefore decrease the time required for the +patch be included in the next version. + +Alfresco has an [active forum](http://community.alfresco.com/community/ecm) to support +community users of our products. If you have any questions then this is the fastest method +of getting an answer. + +## Raising issues + +If you want to raise an issue then please use the [issue tracker on GitHub](https://github.com/Alfresco/SearchServices/issues). +We may convert these to Jira tickets before working on them, as this is the system we use +internally for tracking development. The Jira project for this codebase is [SEARCH](https://issues.alfresco.com/jira/projects/SEARCH/issues) +and you may want to look here first to see if your issue has previously been encountered.[^jiraaccess] +There are some guidelines for raising a good issue [here](https://hub.alfresco.com/t5/alfresco-content-services-hub/reporting-an-issue/ba-p/289727). + +[^jiraaccess]: Note that while we try to keep our Jira issues visible to everyone, some +are restricted as they relate to specific customers or security issues. Some older issues +are also restricted simply because we have not been through to check if they contain +sensitive information or not. + +## Submitting changes + +We have a [coding standards guidelines page](https://hub.alfresco.com/t5/alfresco-content-services-hub/coding-standards-for-alfresco-content-services/ba-p/290457) +although you will find numerous examples where we have not adhered to them. Please try to +maintain consistency with the guidelines for new code, but avoid reformatting large +blocks of code if these are not related to your change. + +## Branches + +Our codebase consists of long-lived release branches and short-lived feature branches. The +code that we expect to include in the next minor version is stored on `master`. All other +release branches have the prefix `release/`. Feature branches may have any other prefix, +but we usually use `feature/` or `fix/`. We expect code on release branches to be ready +to release, and in the rare occasion when a release branch is broken then we try to revert +changes to fix the branch as soon as possible. + +As bug fixes often also need a change to ACS then we use a cherry-pick strategy to get the +fix to all necessary release branches. The fix should initially be merged to `master` and +it can then be cherry-picked back by using: + +```git cherry-pick -x -m 1 [mergeCommitId]``` + +## Community Mirror + +Pull requests to our community mirror will be accepted in our enterprise codebase and then +mirrored back to the community. You will always be credited with your commits, although if +you [sign your commits](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) then the +signature will be stripped by the mirroring process.[^dependabot] + +[^dependabot]: This is the reason that pull requests submitted by Dependabot appear closed +rather than merged. + +## Builds + +Our builds are currently in our internal Bamboo instance. If you have access to the +internal code then you can find some more links in the [insight engine module](insight-engine/CONTRIBUTING.md). + +Although the build results are not visible externally, it should be possible to run most of +the tests locally. We have divided our tests into unit tests, integration tests and +end-to-end tests. The unit and integration tests can be run using the maven `test` and +`verify` goals respectively. The end-to-end tests cannot currently be run externally as +they require some dependencies stored in our internal Nexus. diff --git a/README.md b/README.md index 0b78fd2e7..b973c9523 100644 --- a/README.md +++ b/README.md @@ -41,3 +41,7 @@ More details are available at [search-services](/search-services) folder. **Following resources will not be available for Community users** More details are available at [insight-engine](/insight-engine) folder. + +## Contributing guide + +Please use [this guide](CONTRIBUTING.md) to make a contribution to the project. diff --git a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/.env b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/.env index b39fd3b1e..352f5b342 100755 --- a/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/.env +++ b/e2e-test/generator-alfresco-docker-compose/generators/app/templates/6.2/.env @@ -1,15 +1,15 @@ ALFRESCO_TAG=latest -SHARE_TAG=6.1.0-RC3 -POSTGRES_TAG=10.9 -TRANSFORM_ROUTER_TAG=1.1.0-EA2 -PDF_RENDERER_TAG=2.1.0-EA4 -IMAGE_MAGICK_TAG=2.1.0-EA4 -LIBREOFFICE_TAG=2.1.0-EA4 -TIKA_TAG=2.1.0-EA4 -TRANSFORM_MISC_TAG=2.1.0-EA4 +SHARE_TAG=6.2.0 +POSTGRES_TAG=11.4 +TRANSFORM_ROUTER_TAG=1.1.0 +PDF_RENDERER_TAG=2.1.0 +IMAGE_MAGICK_TAG=2.1.0 +LIBREOFFICE_TAG=2.1.0 +TIKA_TAG=2.1.0 +TRANSFORM_MISC_TAG=2.1.0 SHARED_FILE_STORE_TAG=0.5.3 ACTIVE_MQ_TAG=5.15.8 -DIGITAL_WORKSPACE_TAG=1.1.0 +DIGITAL_WORKSPACE_TAG=1.3.0 ACS_NGINX_TAG=3.0.1 ACS_COMMUNITY_NGINX_TAG=1.0.0 SEARCH_TAG=latest diff --git a/e2e-test/pom.xml b/e2e-test/pom.xml index 5cee7b842..1e48fef17 100644 --- a/e2e-test/pom.xml +++ b/e2e-test/pom.xml @@ -11,10 +11,11 @@ Search Analytics E2E Tests Test Project to test Search Service and Analytics Features on a complete setup of Alfresco, Share - 6.0.1.2 - 6.0.0.4 - 3.0.14 - 3.2.0 + 1.28 + 1.26 + 1.13 + 3.0.18 + 3.3.0 src/test/resources/SearchSuite.xml @@ -44,7 +45,7 @@ org.alfresco.tas - restapi-test + restapi ${tas.rest.api.version} test @@ -56,7 +57,7 @@ org.alfresco.tas - cmis-test + cmis ${tas.cmis.api.version} test @@ -71,6 +72,12 @@ alfresco-governance-services-automation-enterprise-rest-api ${rm.version} test + + + org.alfresco.tas + restapi-test + + com.fasterxml.jackson.core @@ -88,6 +95,10 @@ com.fasterxml.jackson.core jackson-databind + + org.alfresco.tas + restapi-test + diff --git a/e2e-test/src/main/java/org/alfresco/search/TestGroup.java b/e2e-test/src/main/java/org/alfresco/search/TestGroup.java index a149ab6f7..fbb90a64e 100644 --- a/e2e-test/src/main/java/org/alfresco/search/TestGroup.java +++ b/e2e-test/src/main/java/org/alfresco/search/TestGroup.java @@ -38,6 +38,7 @@ public class TestGroup public static final String ACS_61n = "ACS_61n"; // Alfresco Content Services 6.1 or above public static final String ACS_611n = "ACS_611n"; // Alfresco Content Services 6.1.1 or above public static final String ACS_62n = "ACS_62n"; // Alfresco Content Services 6.2 or above + public static final String ACS_63n = "ACS_63n"; // Alfresco Content Services 6.3 or above public static final String AGS_302 = "AGS_302"; // Alfresco governance Services 3.0.2 or above diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/AbstractE2EFunctionalTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/AbstractE2EFunctionalTest.java index 7470000ab..8073d003d 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/AbstractE2EFunctionalTest.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/AbstractE2EFunctionalTest.java @@ -6,6 +6,16 @@ */ package org.alfresco.test.search.functional; +import static java.util.Optional.ofNullable; + +import static lombok.AccessLevel.PROTECTED; +import static org.testng.Assert.assertEquals; + +import java.util.List; +import java.util.Set; +import java.util.stream.Collectors; + +import lombok.Getter; import org.alfresco.cmis.CmisWrapper; import org.alfresco.dataprep.ContentService; import org.alfresco.dataprep.SiteService.Visibility; @@ -38,15 +48,10 @@ import org.springframework.test.context.testng.AbstractTestNGSpringContextTests; import org.testng.annotations.BeforeClass; import org.testng.annotations.BeforeSuite; -import lombok.Getter; - -import static java.util.Optional.ofNullable; -import static lombok.AccessLevel.PROTECTED; - /** * @author meenal bhave */ -@ContextConfiguration("classpath:alfresco-search-e2e-context.xml") +@ContextConfiguration ("classpath:alfresco-search-e2e-context.xml") public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringContextTests { /** The number of retries that a query will be tried before giving up. */ @@ -76,26 +81,26 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont protected CmisWrapper cmisApi; @Autowired - // @Getter(value = PROTECTED) + // @Getter(value = PROTECTED) protected DataUser dataUser; @Autowired - @Getter(value = PROTECTED) + @Getter (value = PROTECTED) private ContentService contentService; protected UserModel testUser, adminUserModel, testUser2; protected SiteModel testSite, testSite2; protected static String unique_searchString; - + protected static final String SEARCH_LANGUAGE_CMIS = "cmis"; - + protected enum SearchLanguage { CMIS, AFTS - } + } - @BeforeSuite(alwaysRun = true) + @BeforeSuite (alwaysRun = true) public void beforeSuite() throws Exception { super.springTestContextPrepareTestInstance(); @@ -104,7 +109,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont deployCustomModel("model/finance-model.xml"); } - @BeforeClass(alwaysRun = true) + @BeforeClass (alwaysRun = true) public void setup() { serverHealth.assertServerIsOnline(); @@ -209,7 +214,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont customModel.setNodeRef(modelInRepo.getId()); customModel.setNodeRef(customModel.getNodeRefWithoutVersion()); customModel.setCmisLocation(String.format("/Data Dictionary/Models/%s", fileName)); - LOGGER.info("Custom Model file: " + customModel.getCmisLocation()); + LOGGER.info("Custom Model file: " + customModel.getCmisLocation()); } else { @@ -239,9 +244,9 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Wait for Solr to finish indexing and search to return appropriate results - * - * @param userQuery Search Query - * @param contentToFind that's expected to be included / excluded from the results + * + * @param userQuery Search Query + * @param contentToFind that's expected to be included / excluded from the results * @param expectedInResults Whether we expect the content in the results or not. * @return true if search returns expected results, i.e. is given content is found or excluded from the results */ @@ -277,26 +282,25 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont return false; } - - /** - * Method to check if the contentName is returned in the SearchResponse - * @param response - * @param contentName - * @return - */ - public boolean isContentInSearchResponse(SearchResponse response, String contentName) - { - boolean found = response.getEntries().stream() - .map(entry -> entry.getModel().getName()) - .filter(name -> name.equalsIgnoreCase(contentName) || contentName.isBlank()).count() > 0; - return found; + /** + * Method to check if the contentName is returned in the SearchResponse. + * + * @param response the search response + * @param contentName the text we are using as matching/verifying criteria. + * @return true if if the item with the contentName text is returned in the SearchResponse. + */ + public boolean isContentInSearchResponse(SearchResponse response, String contentName) + { + return response.getEntries().stream() + .map(entry -> entry.getModel().getName()) + .anyMatch(name -> name.equalsIgnoreCase(contentName) || contentName.isBlank()); } /** * Wait for Solr to finish indexing: Indexing has caught up = true if search returns appropriate results - * - * @param userQuery: search query, this can include the fieldname, unique search string will guarantee accurate results + * + * @param userQuery: search query, this can include the fieldname, unique search string will guarantee accurate results * @param expectedInResults, true if entry is expected in the results set * @return true (indexing is finished) if search returns appropriate results */ @@ -308,7 +312,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * waitForIndexing method that matches / waits for filename, metadata to be indexed. - * + * * @param userQuery * @param expectedInResults * @return @@ -321,7 +325,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * waitForIndexing method that matches / waits for content to be indexed, this can take longer than metadata indexing. * Since Metadata is indexed first, use this method where tests, queries need content to be indexed too. - * + * * @param userQuery * @param expectedInResults * @return @@ -333,9 +337,9 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Wait for Solr to finish indexing: Indexing has caught up = true if search returns appropriate results - * - * @param fieldName: specific field to search for, e.g. name. When specified, the query will become: name:'userQuery' - * @param userQuery: search string, unique search string will guarantee accurate results + * + * @param fieldName: specific field to search for, e.g. name. When specified, the query will become: name:'userQuery' + * @param userQuery: search string, unique search string will guarantee accurate results * @param expectedInResults, true if entry is expected in the results set * @return true (indexing is finished) if search returns appropriate results */ @@ -348,7 +352,7 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Run a search as admin user and return the response - * + * * @param queryString: string to search for, unique search string will guarantee accurate results * @return the search response from the API */ @@ -359,8 +363,8 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Run a search as given user and return the response - * - * @param user: UserModel for the user you wish to run the query as + * + * @param user: UserModel for the user you wish to run the query as * @param queryString: string to search for, unique search string will guarantee accurate results * @return the search response from the API */ @@ -372,11 +376,11 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont searchRequest.setQuery(queryModel); return restClient.authenticateUser(user).withSearchAPI().search(searchRequest); } - + /** * Run a search with Spellcheck as given user and return the response - * @param user UserModel for the user you wish to run the query as - * @param queryModel The queryModel to search for, containing the query + * @param user UserModel for the user you wish to run the query as + * @param queryModel The queryModel to search for, containing the query * @param spellcheckQuery The Spellcheck Model containing the query * @return the search response from the API */ @@ -417,32 +421,22 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont query.setQuery(queryReq); return query; } - + protected DataUser getDataUser() { return dataUser; } - + /** * Helper method to test if the search query works and count matches where provided - * @param query: AFTS or cmis query string - * @param expectedCount: Only successful response is checked, when expectedCount is null (can not be exactly specified), - * @param setCmis: Query language is set to cmis when setCmis is true, AFTS when false + * + * @param query: AFTS or cmis query string + * @param expectedCount: Only successful response is checked, when expectedCount is null (can not be exactly specified), * @return SearchResponse */ protected SearchResponse testSearchQuery(String query, Integer expectedCount, SearchLanguage queryLanguage) { - RestRequestQueryModel queryModel = new RestRequestQueryModel(); - queryModel.setQuery(query); - - if (ofNullable(queryLanguage).isPresent()) - { - queryModel.setLanguage(queryLanguage.toString()); - } - - SearchResponse response = queryAsUser(testUser, queryModel); - - restClient.assertStatusCodeIs(HttpStatus.OK); + SearchResponse response = performSearch(testUser, query, queryLanguage); if (ofNullable(expectedCount).isPresent()) { @@ -451,7 +445,64 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont return response; } + + /** + * Helper method to test if the search query returns the expected results in the order given. + * + * @param query: AFTS or cmis query string + * @param expectedNames: The ordered list of names expected to be returned, + * @return SearchResponse + */ + protected SearchResponse testSearchQueryOrdered(String query, List expectedNames, SearchLanguage queryLanguage) + { + SearchResponse response = performSearch(testUser, query, queryLanguage); + + List names = response.getEntries().stream().map(s -> s.getModel().getName()).collect(Collectors.toList()); + + // Include lists in failure message as TestNG won't do this for lists. + assertEquals(names, expectedNames, "Unexpected results for query: " + query + " Expected: " + expectedNames + " but got " + names); + return response; + } + + /** + * Helper method to test if the search query returns the expected set of results. + * + * @param query: AFTS or cmis query string + * @param expectedNames: The ordered list of names expected to be returned, + * @return SearchResponse + */ + protected SearchResponse testSearchQueryUnordered(String query, Set expectedNames, SearchLanguage queryLanguage) + { + SearchResponse response = performSearch(testUser, query, queryLanguage); + + Set names = response.getEntries().stream().map(s -> s.getModel().getName()).collect(Collectors.toSet()); + + assertEquals(names, expectedNames, "Unexpected results for query: " + query); + + return response; + } + + private SearchResponse performSearch(UserModel asUser, String query, SearchLanguage queryLanguage) + { + RestRequestQueryModel queryModel = new RestRequestQueryModel(); + queryModel.setQuery(query); + + if (!ofNullable(asUser).isPresent()) + { + asUser = testUser; + } + + if (ofNullable(queryLanguage).isPresent()) + { + queryModel.setLanguage(queryLanguage.toString()); + } + + SearchResponse response = queryAsUser(asUser, queryModel); + + return response; + } + /** * Method to create and run a simple spellcheck query * When a spellcheck query is run a user, the query inputed and the user query is inputted @@ -463,8 +514,8 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont { RestRequestSpellcheckModel spellCheck = new RestRequestSpellcheckModel(); spellCheck.setQuery(userQuery); - - UserModel searchUser = ofNullable(user).isPresent()? user: testUser; + + UserModel searchUser = ofNullable(user).isPresent() ? user : testUser; SearchRequest searchReq = new SearchRequest(); RestRequestQueryModel queryReq = new RestRequestQueryModel(); queryReq.setQuery(query); @@ -477,8 +528,8 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont /** * Method to check the spellcheck object returned in the Search Response - * @param response SearchResponse - * @param spellCheckType String Values: searchInsteadFor, didYouMean or null + * @param response SearchResponse + * @param spellCheckType String Values: searchInsteadFor, didYouMean or null * @param spellCheckSuggestion String Values: suggestion string or null */ public void testSearchSpellcheckResponse(SearchResponse response, String spellCheckType, String spellCheckSuggestion) @@ -497,6 +548,6 @@ public abstract class AbstractE2EFunctionalTest extends AbstractTestNGSpringCont { response.getContext().assertThat().field("spellCheck").isNotEmpty(); response.getContext().getSpellCheck().assertThat().field("suggestions").contains(spellCheckSuggestion); - } + } } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPropertyTests.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPropertyTests.java index 40ea49483..d9f7638bd 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPropertyTests.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/cmis/SolrSearchByPropertyTests.java @@ -1,5 +1,8 @@ package org.alfresco.test.search.functional.searchServices.cmis; +import java.util.List; +import java.util.Set; + import org.alfresco.utility.Utility; import org.alfresco.utility.data.CustomObjectTypeProperties; import org.alfresco.utility.data.provider.XMLDataConfig; @@ -15,7 +18,7 @@ public class SolrSearchByPropertyTests extends AbstractCmisE2ETest private FolderModel guestf, tesf, restf, testtttf, testf, testf1, testf2, testf3, testf4; private FileModel guestc, restc, tesc, testtttc, testc, testc1, testc2, testc3; - @BeforeClass(alwaysRun = true) + @BeforeClass (alwaysRun = true) public void dataPreparation() throws Exception { dataContent.usingAdmin().deployContentModel("model/tas-model.xml"); @@ -46,68 +49,464 @@ public class SolrSearchByPropertyTests extends AbstractCmisE2ETest // Sites dataContent.usingUser(testUser).usingSite(testSite).createCustomContent(guestf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "guestf text") - .addProperty("tas:IntPropertyF", 222)); + .addProperty("tas:IntPropertyF", 222)); dataContent.usingUser(testUser).usingSite(testSite).createCustomContent(tesf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "tesf text") - .addProperty("tas:IntPropertyF", 224)); + .addProperty("tas:IntPropertyF", 224)); // Sites >> Folders dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(restf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "restf text") - .addProperty("tas:IntPropertyF", 223)); + .addProperty("tas:IntPropertyF", 223)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testtttf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testtttf text") - .addProperty("tas:IntPropertyF", 225)); + .addProperty("tas:IntPropertyF", 225)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf text") - .addProperty("tas:IntPropertyF", 226)); + .addProperty("tas:IntPropertyF", 226)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf1, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf1 text") - .addProperty("tas:IntPropertyF", 2221)); + .addProperty("tas:IntPropertyF", 2221)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf2, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf2 text") - .addProperty("tas:IntPropertyF", 2222)); + .addProperty("tas:IntPropertyF", 2222)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf3, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf3 text") - .addProperty("tas:IntPropertyF", 2223)); + .addProperty("tas:IntPropertyF", 2223)); dataContent.usingUser(testUser).usingResource(guestf).createCustomContent(testf4, "F:tas:folder", new CustomObjectTypeProperties().addProperty("tas:TextPropertyF", "testf4 text") - .addProperty("tas:IntPropertyF", 2224)); + .addProperty("tas:IntPropertyF", 2224)); // Sites >> Files dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(guestc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "guestc text") - .addProperty("tas:IntPropertyC", 222)); + .addProperty("tas:IntPropertyC", 222)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(restc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "restc text") - .addProperty("tas:IntPropertyC", 223)); + .addProperty("tas:IntPropertyC", 223)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(tesc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "tesc text") - .addProperty("tas:IntPropertyC", 224)); + .addProperty("tas:IntPropertyC", 224)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testtttc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testtttc text") - .addProperty("tas:IntPropertyC", 225)); + .addProperty("tas:IntPropertyC", 225)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testc, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testc text") - .addProperty("tas:IntPropertyC", 226)); + .addProperty("tas:IntPropertyC", 226)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testc1, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testc1 text") - .addProperty("tas:IntPropertyC", 2221)); + .addProperty("tas:IntPropertyC", 2221)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testc2, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testc2 text") - .addProperty("tas:IntPropertyC", 2222)); + .addProperty("tas:IntPropertyC", 2222)); dataContent.usingUser(testUser).usingResource(tesf).createCustomContent(testc3, "D:tas:document", new CustomObjectTypeProperties().addProperty("tas:TextPropertyC", "testc3 text") - .addProperty("tas:IntPropertyC", 2223)); + .addProperty("tas:IntPropertyC", 2223)); // wait for solr index Utility.waitToLoopTime(getSolrWaitTimeInSeconds()); } - @Test(dataProviderClass = XMLTestDataProvider.class, dataProvider = "getQueriesData") - @XMLDataConfig(file = "src/test/resources/testdata/search-by-property.xml") - public void executeSearchByProperty(QueryModel query) throws Exception + @Test + public void testFileNameEquality() { - cmisApi.authenticateUser(testUser).withQuery(query.getValue()).assertResultsCount().equals(query.getResults()); + String query = "SELECT * FROM tas:document where cmis:name = 'testc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt")); + } + + @Test + public void testFileNameInequality() + { + String query = "SELECT * FROM tas:document where cmis:name <> 'testc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "restc.txt", "tesc.txt", "testtttc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileNameIn() + { + String query = "SELECT * FROM tas:document where cmis:name IN('testc.txt', 'guestc.txt', 'restc.txt')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt", "guestc.txt", "restc.txt")); + } + + @Test + public void testFileNameNotIn() + { + // Nb. "gustc" is missing an "e". + String query = "SELECT * FROM tas:document where cmis:name NOT IN('testc.txt', 'gustc.txt', 'restc.txt')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "tesc.txt", "testtttc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileNameLike() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE '%testc%'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileNameLikeExact() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE 'testc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt")); + } + + @Test + public void testFileNamePrefixSuffix() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE 't%tc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt", "testtttc.txt")); + } + + @Test + public void testFileNameUnderscore() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE 't__tc.txt'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt")); + } + + @Test + public void testFolderNameEquality() + { + String query = "SELECT * FROM tas:folder where cmis:name = 'testf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf")); + } + + @Test + public void testFolderNameInequality() + { + String query = "SELECT * FROM tas:folder where cmis:name <> 'testf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "restf", "testtttf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderNameIn() + { + String query = "SELECT * FROM tas:folder where cmis:name IN('testf', 'guestf', 'restf')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf", "guestf", "restf")); + } + + @Test + public void testFolderNameNotIn() + { + // Nb. "gustc" is missing an "e". + String query = "SELECT * FROM tas:folder where cmis:name NOT IN('testf', 'gustf', 'restf')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "testtttf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderNameLike() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE '%testf%'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderNameLikeExact() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE 'testf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf")); + } + + @Test + public void testFolderNamePrefixSuffix() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE 't%tf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttf", "testf")); + } + + @Test + public void testFolderNameUnderscore() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE 't__tf'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf")); + } + + @Test + public void testFileNameOrderAsc() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE '%testc%' ORDER BY cmis:name ASC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testc1.txt", "testc2.txt", "testc3.txt", "testc.txt")); + } + + @Test + public void testFileNameOrderDesc() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE '%testc%' ORDER BY cmis:name DESC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testc.txt", "testc3.txt", "testc2.txt", "testc1.txt")); + } + + @Test + public void testFileOrderNameDescDateAsc() + { + String query = "SELECT * FROM tas:document where cmis:name LIKE '%testc%' ORDER BY cmis:name DESC, cmis:creationDate ASC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testc.txt", "testc3.txt", "testc2.txt", "testc1.txt")); + } + + @Test + public void testFolderNameOrderAsc() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE '%testf%' ORDER BY cmis:name ASC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderNameOrderDesc() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE '%testf%' ORDER BY cmis:name DESC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testf4", "testf3", "testf2", "testf1", "testf")); + } + + @Test + public void testFolderOrderNameDescDateAsc() + { + String query = "SELECT * FROM tas:folder where cmis:name LIKE '%testf%' ORDER BY cmis:name DESC, cmis:creationDate ASC"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningOrderedValues("cmis:name", + List.of("testf4", "testf3", "testf2", "testf1", "testf")); + } + + @Test + public void testFileCustomPropertyEquality() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC = 'restc text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restc.txt")); + } + + @Test + public void testFileCustomPropertyInequality() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC <> 'testc1 text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "restc.txt", "tesc.txt", "testtttc.txt", "testc.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileCustomPropertyIn() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC IN('restc text', 'testc2 text', 'testc3 text')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restc.txt", "testc2.txt", "testc3.txt")); + } + + + @Test + public void testFileCustomPropertyNotIn() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC NOT IN('restc text', 'testc2 text', 'testc3 text')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "tesc.txt", "testtttc.txt", "testc.txt", "testc1.txt")); + } + + @Test + public void testFileCustomPropertyLike() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC LIKE '%restc%'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restc.txt")); + } + + @Test + public void testFileCustomPropertyUnderscore() + { + String query = "SELECT * FROM tas:document where tas:TextPropertyC LIKE 't__tc text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testc.txt")); + } + + @Test + public void testFolderCustomPropertyEquality() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF = 'restf text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restf")); + } + + @Test + public void testFolderCustomPropertyInequality() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF <> 'testf1 text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "restf", "testtttf", "testf", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderCustomPropertyIn() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF IN('restf text', 'testf2 text', 'testf3 text')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restf", "testf2", "testf3")); + } + + @Test + public void testFolderCustomPropertyNotIn() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF NOT IN('restf text', 'testf2 text', 'testf3 text')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "testtttf", "testf", "testf1", "testf4")); + } + + @Test + public void testFolderCustomPropertyLike() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF LIKE '%restf%'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("restf")); + } + + @Test + public void testFolderCustomPropertyUnderscore() + { + String query = "SELECT * FROM tas:folder where tas:TextPropertyF LIKE 't__tf text'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testf")); + } + + @Test + public void testFileIntEquality() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC = '222'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt")); + } + + @Test + public void testFileIntInequality() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC <> '223'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "tesc.txt", "testtttc.txt", "testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileIntLessThen() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC < '223'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt")); + } + + @Test + public void testFileIntLessThanOrEqual() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC <= '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "restc.txt", "tesc.txt")); + } + + @Test + public void testFileIntGreaterThanOrEqual() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC >= '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("tesc.txt", "testtttc.txt", "testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileIntGreaterThan() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC > '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttc.txt", "testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFileIntIn() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC IN('222', '223', '224', '225')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestc.txt", "restc.txt", "tesc.txt", "testtttc.txt")); + } + + @Test + public void testFileIntNotIn() + { + String query = "SELECT * FROM tas:document where tas:IntPropertyC NOT IN('222', '223', '224')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttc.txt", "testc.txt", "testc1.txt", "testc2.txt", "testc3.txt")); + } + + @Test + public void testFolderIntEquality() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF = '222'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf")); + } + + @Test + public void testFolderIntInequality() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF <> '223'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "testtttf", "testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderIntLessThan() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF < '223'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf")); + } + + @Test + public void testFolderIntLessThanOrEqual() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF <= '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "restf")); + } + + @Test + public void testFolderIntGreaterThanOrEqual() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF >= '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("tesf", "testtttf", "testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderIntGreaterThan() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF > '224'"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttf", "testf", "testf1", "testf2", "testf3", "testf4")); + } + + @Test + public void testFolderIntIn() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF IN('222', '223', '224', '225')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("guestf", "tesf", "restf", "testtttf")); + } + + @Test + public void testFolderIntNotIn() + { + String query = "SELECT * FROM tas:folder where tas:IntPropertyF NOT IN('222', '223', '224')"; + cmisApi.authenticateUser(testUser).withQuery(query).assertValues().isReturningValues("cmis:name", + Set.of("testtttf", "testf", "testf1", "testf2", "testf3", "testf4")); } } diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/FacetRangeSearchTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/FacetRangeSearchTest.java index 4dfc0c2b9..5b52698a2 100644 --- a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/FacetRangeSearchTest.java +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/FacetRangeSearchTest.java @@ -22,9 +22,7 @@ import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.core.Is.is; import static org.testng.Assert.assertEquals; import static org.testng.Assert.assertNull; -import static org.testng.Assert.assertTrue; -import java.util.ArrayList; import java.util.List; import java.util.Map; @@ -66,8 +64,8 @@ import org.testng.annotations.Test; * "gap": "+100DAY" * } * } - * @author Michael Suzuki * + * @author Michael Suzuki */ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { @@ -78,49 +76,37 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest waitForContentIndexing(file4.getContent(), true); } + /** Check the error messages mention the mandatory fields when they are omitted. */ @Test @TestRail(section = {TestGroup.REST_API, TestGroup.SEARCH }, executionType = ExecutionType.REGRESSION, description = "Check facet intervals mandatory fields") public void checkingFacetsMandatoryErrorMessages() { SearchRequest query = createQuery("cars"); - List ranges = new ArrayList<>(); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - ranges.add(facetRangeModel); - query.setRanges(ranges); + + // Omit the field. + query.setRanges(List.of(createRangesModel(null, "0", "400", "20"))); query(query); - restClient.assertStatusCodeIs(HttpStatus.BAD_REQUEST).assertLastError() - .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "field")); - ranges.clear(); - facetRangeModel.setField("content.size"); - ranges.add(facetRangeModel); - query.setRanges(ranges); + .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "field")); + // Omit the start. + query.setRanges(List.of(createRangesModel("content.size", null, "400", "20"))); query(query); - restClient.assertStatusCodeIs(HttpStatus.BAD_REQUEST).assertLastError() - .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "start")); - facetRangeModel.setStart("0"); - ranges.clear(); - ranges.add(facetRangeModel); - query.setRanges(ranges); + .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "start")); + // Omit the end. + query.setRanges(List.of(createRangesModel("content.size", "0", null, "20"))); query(query); - restClient.assertStatusCodeIs(HttpStatus.BAD_REQUEST).assertLastError() - .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "end")); - facetRangeModel.setEnd("400"); - query.setRanges(ranges); - ranges.clear(); - ranges.add(facetRangeModel); + .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "end")); + // Omit the gap. + query.setRanges(List.of(createRangesModel("content.size", "0", "400", null))); query(query); - restClient.assertStatusCodeIs(HttpStatus.BAD_REQUEST).assertLastError() - .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "gap")); - - facetRangeModel.setGap("100"); + .containsSummary(String.format(RestErrorModel.MANDATORY_PARAM, "gap")); } @Test @@ -131,13 +117,8 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { SearchRequest query = createQuery("* AND SITE:'" + testSite.getId() + "'"); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("content.size"); - facetRangeModel.setStart("0"); - facetRangeModel.setEnd("200"); - facetRangeModel.setGap("20"); - List ranges = new ArrayList<>(); - ranges.add(facetRangeModel); + RestRequestRangesModel facetRangeModel = createRangesModel("content.size", "0", "200", "20"); + List ranges = List.of(facetRangeModel); query.setRanges(ranges); SearchResponse response = query(query); response.assertThat().entriesListIsNotEmpty(); @@ -147,14 +128,14 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest RestGenericBucketModel bucket = facetResponseModel.getBuckets().get(0); bucket.assertThat().field("label").is("[20 - 40)"); bucket.assertThat().field("filterQuery").is("content.size:[\"20\" TO \"40\">"); - Map metric = (Map) bucket.getMetrics().get(0).getValue(); + Map metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 2, "Unexpected count for first bucket."); Map info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"20"); - assertEquals(info.get("end"),"40"); + assertEquals(info.get("start"), "20"); + assertEquals(info.get("end"), "40"); assertNull(info.get("count")); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"false"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "false"); bucket = facetResponseModel.getBuckets().get(1); bucket.assertThat().field("label").is("[40 - 120)"); @@ -162,20 +143,20 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for second bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"40"); - assertEquals(info.get("end"),"120"); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"false"); + assertEquals(info.get("start"), "40"); + assertEquals(info.get("end"), "120"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "false"); bucket = facetResponseModel.getBuckets().get(2); bucket.assertThat().field("label").is("[120 - 200]"); bucket.assertThat().field("filterQuery").is("content.size:[\"120\" TO \"200\"]"); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for third bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"120"); - assertEquals(info.get("end"),"200"); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("start"), "120"); + assertEquals(info.get("end"), "200"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "true"); } @Test @@ -186,14 +167,9 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { SearchRequest query = createQuery("* AND SITE:'" + testSite.getId() + "'"); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("content.size"); - facetRangeModel.setStart("0"); - facetRangeModel.setEnd("200"); - facetRangeModel.setGap("20"); + RestRequestRangesModel facetRangeModel = createRangesModel("content.size", "0", "200", "20"); facetRangeModel.setHardend(true); - List ranges = new ArrayList<>(); - ranges.add(facetRangeModel); + List ranges = List.of(facetRangeModel); query.setRanges(ranges); SearchResponse response = query(query); response.assertThat().entriesListIsNotEmpty(); @@ -203,26 +179,26 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest RestGenericBucketModel bucket = facetResponseModel.getBuckets().get(0); bucket.assertThat().field("label").is("[20 - 40)"); bucket.assertThat().field("filterQuery").is("content.size:[\"20\" TO \"40\">"); - Map metric = (Map) bucket.getMetrics().get(0).getValue(); + Map metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 2, "Unexpected count for first bucket."); Map info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"20"); - assertEquals(info.get("end"),"40"); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"false"); + assertEquals(info.get("start"), "20"); + assertEquals(info.get("end"), "40"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "false"); assertNull(info.get("count")); bucket = facetResponseModel.getBuckets().get(1); bucket.assertThat().field("label").is("[40 - 120)"); bucket.assertThat().field("filterQuery").is("content.size:[\"40\" TO \"120\">"); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"40"); - assertEquals(info.get("end"),"120"); + assertEquals(info.get("start"), "40"); + assertEquals(info.get("end"), "120"); metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for second bucket."); assertNull(info.get("count")); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"false"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "false"); bucket = facetResponseModel.getBuckets().get(2); bucket.assertThat().field("label").is("[120 - 200]"); @@ -230,11 +206,11 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for third bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"120"); - assertEquals(info.get("end"),"200"); + assertEquals(info.get("start"), "120"); + assertEquals(info.get("end"), "200"); assertNull(info.get("count")); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "true"); } /** This test relies on a document created in 2015 existing, probably part of the sample site. */ @@ -246,13 +222,8 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { SearchRequest query = createQuery("name:A*"); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("created"); - facetRangeModel.setStart("2015-09-29T10:45:15.729Z"); - facetRangeModel.setEnd("2016-09-29T10:45:15.729Z"); - facetRangeModel.setGap("+280DAY"); - List ranges = new ArrayList<>(); - ranges.add(facetRangeModel); + RestRequestRangesModel facetRangeModel = createRangesModel("created", "2015-09-29T10:45:15.729Z", "2016-09-29T10:45:15.729Z", "+280DAY"); + List ranges = List.of(facetRangeModel); query.setRanges(ranges); SearchResponse response = query(query); response.assertThat().entriesListIsNotEmpty(); @@ -260,18 +231,18 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest RestGenericFacetResponseModel facetResponseModel = response.getContext().getFacets().get(0); List buckets = facetResponseModel.getBuckets(); - assertThat(buckets.size(),is(1)); + assertThat(buckets.size(), is(1)); RestGenericBucketModel bucket = buckets.get(0); bucket.assertThat().field("label").is("[2015-09-29T10:45:15.729Z - 2017-04-11T10:45:15.729Z]"); bucket.assertThat().field("filterQuery").is("created:[\"2015-09-29T10:45:15.729Z\" TO \"2017-04-11T10:45:15.729Z\"]"); bucket.getMetrics().get(0).assertThat().field("value").is("{count=1}"); Map info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"2015-09-29T10:45:15.729Z"); - assertEquals(info.get("end"),"2017-04-11T10:45:15.729Z"); - assertNull(info.get("count"),"1"); - assertEquals(info.get("startInclusive"),"true"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("start"), "2015-09-29T10:45:15.729Z"); + assertEquals(info.get("end"), "2017-04-11T10:45:15.729Z"); + assertNull(info.get("count"), "1"); + assertEquals(info.get("startInclusive"), "true"); + assertEquals(info.get("endInclusive"), "true"); } @Test @@ -280,19 +251,9 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest public void searchDateAndSizeRanges() { SearchRequest query = createQuery("* AND SITE:'" + testSite.getId() + "'"); - List ranges = new ArrayList<>(); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("created"); - facetRangeModel.setStart("2015-09-29T10:45:15.729Z"); - facetRangeModel.setEnd("2016-09-29T10:45:15.729Z"); - facetRangeModel.setGap("+280DAY"); - ranges.add(facetRangeModel); - RestRequestRangesModel facetCountRangeModel = new RestRequestRangesModel(); - facetCountRangeModel.setField("content.size"); - facetCountRangeModel.setStart("0"); - facetCountRangeModel.setEnd("500"); - facetCountRangeModel.setGap("200"); - ranges.add(facetCountRangeModel); + RestRequestRangesModel facetRangeModel = createRangesModel("created", "2015-09-29T10:45:15.729Z", "2016-09-29T10:45:15.729Z", "+280DAY"); + RestRequestRangesModel facetCountRangeModel = createRangesModel("content.size", "0", "500", "200"); + List ranges = List.of(facetRangeModel, facetCountRangeModel); query.setRanges(ranges); } @@ -304,16 +265,10 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest { SearchRequest query = createQuery("* AND SITE:'" + testSite.getId() + "'"); - RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); - facetRangeModel.setField("content.size"); - facetRangeModel.setStart("0"); - facetRangeModel.setEnd("200"); - facetRangeModel.setGap("20"); - List include = new ArrayList<>(); - include.add("upper"); + RestRequestRangesModel facetRangeModel = createRangesModel("content.size", "0", "200", "20"); + List include = List.of("upper"); facetRangeModel.setInclude(include); - List ranges = new ArrayList<>(); - ranges.add(facetRangeModel); + List ranges = List.of(facetRangeModel); query.setRanges(ranges); SearchResponse response = query(query); response.assertThat().entriesListIsNotEmpty(); @@ -323,14 +278,14 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest RestGenericBucketModel bucket = facetResponseModel.getBuckets().get(0); bucket.assertThat().field("label").is("(20 - 40]"); bucket.assertThat().field("filterQuery").is("content.size:<\"20\" TO \"40\"]"); - Map metric = (Map) bucket.getMetrics().get(0).getValue(); + Map metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 2, "Unexpected count for first bucket."); Map info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"20"); - assertEquals(info.get("end"),"40"); + assertEquals(info.get("start"), "20"); + assertEquals(info.get("end"), "40"); assertNull(info.get("count")); - assertEquals(info.get("startInclusive"),"false"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("startInclusive"), "false"); + assertEquals(info.get("endInclusive"), "true"); bucket = facetResponseModel.getBuckets().get(1); bucket.assertThat().field("label").is("(40 - 120]"); @@ -338,10 +293,10 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for second bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"40"); - assertEquals(info.get("end"),"120"); - assertEquals(info.get("startInclusive"),"false"); - assertEquals(info.get("endInclusive"),"true"); + assertEquals(info.get("start"), "40"); + assertEquals(info.get("end"), "120"); + assertEquals(info.get("startInclusive"), "false"); + assertEquals(info.get("endInclusive"), "true"); bucket = facetResponseModel.getBuckets().get(2); bucket.assertThat().field("label").is("(120 - 200]"); @@ -349,9 +304,28 @@ public class FacetRangeSearchTest extends AbstractSearchServicesE2ETest metric = (Map) bucket.getMetrics().get(0).getValue(); assertEquals(Integer.valueOf(metric.get("count")).intValue(), 1, "Unexpected count for third bucket."); info = (Map) bucket.getBucketInfo(); - assertEquals(info.get("start"),"120"); - assertEquals(info.get("end"),"200"); - assertEquals(info.get("startInclusive"),"false"); - assertEquals(info.get("endInclusive"),"true"); - } + assertEquals(info.get("start"), "120"); + assertEquals(info.get("end"), "200"); + assertEquals(info.get("startInclusive"), "false"); + assertEquals(info.get("endInclusive"), "true"); + } + + /** + * Create a ranges model with the values given. + * + * @param field The field to facet on. + * @param start The lowest facet value. + * @param end The highest facet value. + * @param gap The size of the buckets. + * @return The facet ranges model. + */ + private RestRequestRangesModel createRangesModel(String field, String start, String end, String gap) + { + RestRequestRangesModel facetRangeModel = new RestRequestRangesModel(); + facetRangeModel.setField(field); + facetRangeModel.setStart(start); + facetRangeModel.setEnd(end); + facetRangeModel.setGap(gap); + return facetRangeModel; + } } \ No newline at end of file diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchAFTSInFieldTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchAFTSInFieldTest.java new file mode 100644 index 000000000..987f141d1 --- /dev/null +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchAFTSInFieldTest.java @@ -0,0 +1,326 @@ +/* + * Copyright 2019 Alfresco Software, Ltd. All rights reserved. + * License rights for this program may be obtained from Alfresco Software, Ltd. + * pursuant to a written agreement and any use of this program without such an + * agreement is prohibited. + */ + +package org.alfresco.test.search.functional.searchServices.search; + +import static java.util.List.of; + +import static jersey.repackaged.com.google.common.collect.Sets.newHashSet; + +import java.util.ArrayList; +import java.util.List; +import java.util.Set; +import java.util.stream.Stream; + +import org.alfresco.rest.model.RestNodeAssociationModelCollection; +import org.alfresco.rest.model.RestNodeChildAssociationModel; +import org.alfresco.rest.search.SearchResponse; +import org.alfresco.search.TestGroup; +import org.alfresco.test.search.functional.AbstractE2EFunctionalTest; +import org.alfresco.utility.model.FileModel; +import org.alfresco.utility.model.FileType; +import org.alfresco.utility.model.FolderModel; +import org.testng.Assert; +import org.testng.annotations.BeforeClass; +import org.testng.annotations.Test; + +/** + * Test class tests AFTS Search In Field works + * Created for Search-840 + * + * @author Meenal Bhave + */ +public class SearchAFTSInFieldTest extends AbstractE2EFunctionalTest +{ + private FolderModel folder1, folder2; + private FileModel file1, file2, file3, file4, file5; + + @BeforeClass(alwaysRun = true) + public void dataPreparation() + { + // Create Folders: + // Folder1: Expected to be found with file.txt + folder1 = new FolderModel("file txt folder"); + dataContent.usingUser(testUser).usingSite(testSite).createFolder(folder1); + + // Folder2: Not expected to be found with file.txt + folder2 = new FolderModel("txt files folder"); + dataContent.usingUser(testUser).usingSite(testSite).createFolder(folder2); + + // Create File(s): Expected to be found with file.txt + file1 = new FileModel("file.txt", "file.txt", "", FileType.TEXT_PLAIN, "file.txt"); + + file2 = new FileModel("1-file.txt", "1-file.txt", "", FileType.TEXT_PLAIN, "1-file.txt"); + + file3 = new FileModel("file1.txt", "file1.txt", "", FileType.TEXT_PLAIN, "file1.txt"); + + file4 = new FileModel("txt file", "txt file", "", FileType.TEXT_PLAIN, "txt file"); + + // Not Expected to be found with file.txt + file5 = new FileModel("txt files", "txt files", "", FileType.TEXT_PLAIN, "txt files"); + + of(file1, file2, file3, file4, file5).forEach( + f -> dataContent.usingUser(testUser).usingSite(testSite).usingResource(folder1).createContent(f)); + + waitForContentIndexing(file5.getContent(), true); + } + + @Test(priority = 1, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldName() + { + // Field names in various formats + Stream fieldNames = Stream.of("{http://www.alfresco.org/model/content/1.0}name", + "@{http://www.alfresco.org/model/content/1.0}name", + "cm_name", + "cm:name", + "@cm:name", + "name"); + + // For each field name, check that queries return consistent results with / out '' + fieldNames.forEach(fieldName -> + { + // Query string without quotes + String query = fieldName + ":file.txt"; + + Set expectedNames = newHashSet(); + expectedNames.add("file.txt"); // file1 + expectedNames.add("1-file.txt"); // file2 + expectedNames.add("file1.txt"); // file3 + expectedNames.add("txt file"); // file4 + expectedNames.add("file txt folder"); // folder1 + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + // Query string in single quotes + query = fieldName + ":'file.txt'"; + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + }); + } + + @Test(priority = 2, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldTitle() + { + // Field names in various formats + Stream fieldNames = Stream.of("{http://www.alfresco.org/model/content/1.0}title", + "@{http://www.alfresco.org/model/content/1.0}title", + "cm_title", + "cm:title", + "@cm:title"); + + // For each field name, check that queries return consistent results with / out '' + fieldNames.forEach(fieldName -> + { + String query = fieldName + ":" + file2.getName(); + boolean fileFound = isContentInSearchResults(query, file2.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + testSearchQuery(query, 1, SearchLanguage.AFTS); + + query = fieldName + ":'" + file2.getName() + "\'"; + fileFound = isContentInSearchResults(query, file2.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + testSearchQuery(query, 1, SearchLanguage.AFTS); + }); + } + + @Test(priority = 3, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldContent() + { + // Field names in various formats + List fieldNames = new ArrayList<>(); + fieldNames.add("TEXT"); + fieldNames.add("{http://www.alfresco.org/model/dictionary/1.0}content"); + fieldNames.add("cm:content"); + fieldNames.add("d:content"); + + // For each field name, check that queries return consistent results with / out '' + fieldNames.forEach(fieldName -> + { + String query = fieldName + ":" + file3.getContent(); + boolean fileFound = isContentInSearchResults(query, file3.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + Integer resultCount1 = testSearchQuery(query, null, SearchLanguage.AFTS).getPagination().getTotalItems(); + + query = fieldName + ":'" + file3.getContent() + "\'"; + fileFound = isContentInSearchResults(query, file3.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + testSearchQuery(query, resultCount1, SearchLanguage.AFTS).getPagination().getTotalItems(); + }); + } + + @Test(priority = 4) + public void testSearchInFieldSITE() + { + String query = "SITE:" + testSite.getId(); + boolean fileFound = isContentInSearchResults(query, folder1.getName(), true); + Assert.assertTrue(fileFound, "Site Not found for query: " + query); + + Integer resultCount1 = testSearchQuery(query, 8, SearchLanguage.AFTS).getPagination().getTotalItems(); + + query = "SITE:'" + testSite.getId() + "\'"; + fileFound = isContentInSearchResults(query, folder1.getName(), true); + Assert.assertTrue(fileFound, "Site Not found for query: " + query); + + testSearchQuery(query, resultCount1, SearchLanguage.AFTS).getPagination().getTotalItems(); + } + + @Test(priority = 5) + public void testSearchInFieldTYPE() + { + // Field names in various formats + List fieldNames = new ArrayList<>(); + fieldNames.add("TYPE"); + fieldNames.add("EXACTTYPE"); + + // For each field name, check that queries return consistent results with / out '' + fieldNames.forEach(fieldName -> { + + String query = fieldName + ":cm\\:content" + " and =cm:name:" + file1.getName(); + boolean fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "Content Not found for query: " + query); + + Integer resultCount1 = testSearchQuery(query, 1, SearchLanguage.AFTS).getPagination().getTotalItems(); + + query = fieldName + ":'cm:content'" + " and =cm:name:" + file1.getName(); + fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "Content Not found for query: " + query); + + testSearchQuery(query, resultCount1, SearchLanguage.AFTS).getPagination().getTotalItems(); + }); + } + + @Test(priority = 6) + public void testSearchInFieldID() throws Exception + { + String query = "ID:'workspace://SpacesStore/" + file1.getNodeRefWithoutVersion() + "'"; + boolean fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "Content Not found for query: " + query); + + testSearchQuery(query, 1, SearchLanguage.AFTS).getPagination().getTotalItems(); + } + + @Test(priority = 7) + public void testSearchInFieldPARENT() + { + String query = "PARENT:" + folder1.getNodeRefWithoutVersion(); + + Set expectedNames = newHashSet(); + expectedNames.add(file1.getName()); + expectedNames.add(file2.getName()); + expectedNames.add(file3.getName()); + expectedNames.add(file4.getName()); + expectedNames.add(file5.getName()); + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + query = "PARENT:'" + folder1.getNodeRefWithoutVersion() + "\'"; + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + } + + @Test(priority = 8) + public void testSearchInFieldPRIMARYPARENT() throws Exception + { + // Create Secondary association in testFolder2 + RestNodeChildAssociationModel childAssoc1 = new RestNodeChildAssociationModel(file1.getNodeRefWithoutVersion(), "cm:contains"); + String secondaryChildrenBody = "[" + childAssoc1.toJson() + "]"; + + restClient.authenticateUser(testUser).withCoreAPI().usingResource(folder2).createSecondaryChildren(secondaryChildrenBody); + RestNodeAssociationModelCollection secondaryChildren = restClient.authenticateUser(testUser).withCoreAPI().usingResource(folder2).getSecondaryChildren(); + secondaryChildren.getEntryByIndex(0).assertThat().field("id").is(file1.getNodeRefWithoutVersion()); + + String query = "PRIMARYPARENT:'workspace://SpacesStore/" + folder1.getNodeRef() + "'"; + + Set expectedNames = newHashSet(); + expectedNames.add(file1.getName()); + expectedNames.add(file2.getName()); + expectedNames.add(file3.getName()); + expectedNames.add(file4.getName()); + expectedNames.add(file5.getName()); + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + query = "PARENT:'workspace://SpacesStore/" + folder2.getNodeRef() + "'"; + boolean fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "Expected Content Not found for query: " + query); + + testSearchQuery(query, 1, SearchLanguage.AFTS); + } + + @Test(priority = 9, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldNameExactMatch() + { + // Check that queries return consistent results with / out '' + String query = "=name:" + file1.getName(); + boolean fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + Integer resultCount1 = testSearchQuery(query, 1, SearchLanguage.AFTS).getPagination().getTotalItems(); + Assert.assertSame(resultCount1, 1, "File count does not match for query: " + query); + + query = "=name:'" + file1.getName() + "\'"; + fileFound = isContentInSearchResults(query, file1.getName(), true); + Assert.assertTrue(fileFound, "File Not found for query: " + query); + + testSearchQuery(query, resultCount1, SearchLanguage.AFTS).getPagination().getTotalItems(); + } + + @Test(priority = 10, groups = { TestGroup.ACS_63n }) + public void testSearchInFieldNameQueryExpansion() + { + // Check that queries return consistent results with / out '' + String query = "~name:" + file1.getName(); + + Set expectedNames = newHashSet(); + expectedNames.add(file1.getName()); + expectedNames.add(file2.getName()); + expectedNames.add(file3.getName()); + expectedNames.add(file4.getName()); + expectedNames.add(folder1.getName()); + + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + query = "~name:'" + file1.getName() + "\'"; + testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + } + + @Test(priority = 11, groups = { TestGroup.ACS_63n }) + public void testWithConjunctionDisjunctionAndNegation() + { + // Query string to include Conjunction, Disjunction and Negation + + String query1 = "~name:" + file1.getName(); // Query expected to return 5 results + String query2 = "=name:" + file2.getName(); // Query expected to return 1 result + String query3 = "=name:" + file3.getName(); // Query expected to return 1 result + + // Check Query with Conjunction, Negation, Disjunction: returns right results + // "~name:file.txt and ! (=name:1-file.txt or =name:file1.txt)" + String query = query1 + " and ! (" + query2 + " or " + query3 + ")"; + + // Check that expected files are included in the results + Set expectedNames = newHashSet(); + expectedNames.add("file.txt"); // file1 + expectedNames.add("txt file"); // file4 + expectedNames.add("file txt folder"); // folder1 + + SearchResponse response = testSearchQueryUnordered(query, expectedNames, SearchLanguage.AFTS); + + // Check result count is 5-(1+1)=3 + int resultCount = response.getPagination().getTotalItems(); + Assert.assertEquals(resultCount, 3, "File count does not match for query: " + query); + + // Check that file2 and file3 are excluded from the results + boolean fileFound = isContentInSearchResponse(response, file2.getName()); + Assert.assertFalse(fileFound, "File2 found for query: " + query); + + fileFound = isContentInSearchResponse(response, file3.getName()); + Assert.assertFalse(fileFound, "File3 found for query: " + query); + } +} diff --git a/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchSecondaryAssociationTest.java b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchSecondaryAssociationTest.java new file mode 100644 index 000000000..b382bc6f3 --- /dev/null +++ b/e2e-test/src/test/java/org/alfresco/test/search/functional/searchServices/search/SearchSecondaryAssociationTest.java @@ -0,0 +1,91 @@ +/* + * Copyright 2019 Alfresco Software, Ltd. All rights reserved. + * License rights for this program may be obtained from Alfresco Software, Ltd. + * pursuant to a written agreement and any use of this program without such an + * agreement is prohibited. + */ + +package org.alfresco.test.search.functional.searchServices.search; + +import org.alfresco.rest.model.RestNodeAssociationModelCollection; +import org.alfresco.rest.model.RestNodeChildAssociationModel; +import org.alfresco.test.search.functional.AbstractE2EFunctionalTest; +import org.alfresco.utility.data.CustomObjectTypeProperties; +import org.alfresco.utility.model.FileModel; +import org.alfresco.utility.model.FolderModel; +import org.testng.Assert; +import org.testng.annotations.BeforeClass; +import org.testng.annotations.Test; + +/** + * Test class tests content in the secondary parent is found too + * Created for Search-1313 + * + * @author Meenal Bhave + */ +public class SearchSecondaryAssociationTest extends AbstractE2EFunctionalTest +{ + private FolderModel testFolder1, testFolder2; + private FileModel file1; + + @BeforeClass(alwaysRun = true) + public void dataPreparation() + { + // Folders + testFolder1 = new FolderModel("folder1"); + testFolder2 = new FolderModel("folder2"); + + // File(s) + file1 = new FileModel("file1.txt"); + file1.setContent("content file 1"); + + // Create folder1 + dataContent.usingUser(testUser).usingSite(testSite).createCustomContent(testFolder1, "cmis:folder", new CustomObjectTypeProperties()); + + // Create file1 + dataContent.usingUser(testUser).usingResource(testFolder1).createCustomContent(file1, "cmis:document", new CustomObjectTypeProperties()); + + // Create folder2 + dataContent.usingUser(testUser).usingSite(testSite).createCustomContent(testFolder2, "cmis:folder", new CustomObjectTypeProperties()); + + // wait for solr index + waitForMetadataIndexing(file1.getName(), true); + } + + @Test(priority = 1) + public void testSearchPathForSecondaryAssociation() throws Exception + { + String queryPathFolder1 = "PATH:\"/app:company_home/st:sites/cm:" + testSite.getTitle() + + "/cm:documentLibrary/cm:" + testFolder1.getName() + "/cm:" + file1.getName() + "\""; + + // Test if file can be found in folder1: Primary Parent + boolean found = isContentInSearchResults(queryPathFolder1, file1.getName(), true); + Assert.assertTrue(found, "File Not found using Primary Parent Path"); + + String queryPathFolder2 = "PATH:\"/app:company_home/st:sites/cm:" + testSite.getTitle() + + "/cm:documentLibrary/cm:" + testFolder2.getName() + "/cm:" + file1.getName() + "\""; + + // Test if file can not be found in folder2 + found = isContentInSearchResults(queryPathFolder2, file1.getName(), false); + Assert.assertTrue(found, "File found using Secondary Parent Path"); + + // Create Secondary association in folder2 + RestNodeChildAssociationModel childAssoc1 = new RestNodeChildAssociationModel(file1.getNodeRefWithoutVersion(), "cm:contains"); + String secondaryChildrenBody = "[" + childAssoc1.toJson() + "]"; + + restClient.authenticateUser(testUser).withCoreAPI().usingResource(testFolder2).createSecondaryChildren(secondaryChildrenBody); + RestNodeAssociationModelCollection secondaryChildren = restClient.authenticateUser(testUser).withCoreAPI().usingResource(testFolder2).getSecondaryChildren(); + secondaryChildren.getEntryByIndex(0).assertThat().field("id").is(file1.getNodeRefWithoutVersion()); + + // Test if file can be found in folder2: Secondary Parent + found = isContentInSearchResults(queryPathFolder2, file1.getName(), true); + Assert.assertTrue(found, "File Not found using Secondary Parent Path"); + + // Remove Secondary association + restClient.authenticateUser(testUser).withCoreAPI().usingResource(testFolder2).deleteSecondaryChild(secondaryChildren.getEntryByIndex(0)); + + // Test if file can not be found in folder2 + found = isContentInSearchResults(queryPathFolder2, file1.getName(), false); + Assert.assertTrue(found, "File found using Secondary Parent Path"); + } +} diff --git a/e2e-test/src/test/resources/testdata/search-by-property.xml b/e2e-test/src/test/resources/testdata/search-by-property.xml deleted file mode 100644 index 0b845ec67..000000000 --- a/e2e-test/src/test/resources/testdata/search-by-property.xml +++ /dev/null @@ -1,188 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/pom.xml b/pom.xml index 8863b968c..f21eb44c8 100644 --- a/pom.xml +++ b/pom.xml @@ -29,7 +29,7 @@ 11 6.6.5 - ${solr.base.version}-patched.1 + ${solr.base.version}-patched.2 @@ -66,6 +66,9 @@ + + alphabetical + diff --git a/search-services/README.md b/search-services/README.md index f44e980ca..0c82e56d1 100644 --- a/search-services/README.md +++ b/search-services/README.md @@ -338,6 +338,22 @@ This Docker Image is available at Alfresco Docker Hub: To use the public image instead of the local one (`searchservices:develop`) just use `alfresco/alfresco-search-services:1.3.x.x` labels. +## Docker Master-Slave setup +### Enable Search Slave Replica config + +To enable slave node specify environment value `REPLICATION_TYPE=slave`, by default Master config is enabled and slave is disabled. + +During deployment time whenever Search Services or Insight Engine image starts, it will execute the script [search_config_setup.sh](/packaging/src/docker) which will configure the slave config setup based on the value specified in the script. + +To run the docker image: + +```bash +$ docker run -p 8984:8983 -e REPLICATION_TYPE=slave -e ALFRESCO_SECURE_COMMS=none -e SOLR_CREATE_ALFRESCO_DEFAULTS=alfresco,archive searchservices:develop +``` +Solr-slave End point: [http://localhost:8984/solr](http://localhost:8984/solr) + +To generate your own Docker-compose file please follow [generator-alfresco-docker-compose](../e2e-test/generator-alfresco-docker-compose/README.md) + ### Use Alfresco Search Services Docker Image with Docker Compose Sample configuration in a Docker Compose file using **Plain HTTP** protocol to communicate with Alfresco Repository. diff --git a/search-services/alfresco-search/doc/architecture/trackers/00001-content-tracker.md b/search-services/alfresco-search/doc/architecture/trackers/00001-content-tracker.md index e267a79a9..0387374eb 100644 --- a/search-services/alfresco-search/doc/architecture/trackers/00001-content-tracker.md +++ b/search-services/alfresco-search/doc/architecture/trackers/00001-content-tracker.md @@ -155,7 +155,6 @@ The following table illustrates the configuration properties used by the Tracker |shard.method|"DB_ID"|Data (Documents, ACLs) Routing criteria among shards| | |Y|Y| | | |alfresco.fingerprint|true|true if we want to compute the content Fingerprint| | |Y|| | | |alfresco.index.transformContent|true| | | |Y|| | | -|alfresco.version|5.0.0|The target Alfresco version| | | | | | | |alfresco.corePoolSize|4|The number of threads to keep in the pool, even if they are idle|Y|Y|Y|Y|Y|Y| |alfresco.maximumPoolSize|-1|The maximum number of threads allowed in the pool|Y|Y|Y|Y|Y|Y| |alfresco.keepAliveTime|120|When the number of threads is greater than the core pool size, this is the maximum time that excess idle threads will wait for new tasks before terminating|Y|Y|Y|Y|Y|Y| diff --git a/search-services/alfresco-search/pom.xml b/search-services/alfresco-search/pom.xml index 8cdd7d1f7..1bffd3a2b 100644 --- a/search-services/alfresco-search/pom.xml +++ b/search-services/alfresco-search/pom.xml @@ -96,18 +96,69 @@ 2.3.2 + + org.apache.cxf + cxf-core + ${cxf.version} + + + org.apache.cxf + cxf-rt-bindings-soap + ${cxf.version} + + + org.apache.cxf + cxf-rt-bindings-xml + ${cxf.version} + + + org.apache.cxf + cxf-rt-databinding-jaxb + ${cxf.version} + + + org.apache.cxf + cxf-rt-frontend-jaxws + ${cxf.version} + + + org.apache.cxf + cxf-rt-frontend-simple + ${cxf.version} + + + org.apache.cxf + cxf-rt-transports-http + ${cxf.version} + + + org.apache.cxf + cxf-rt-ws-addr + ${cxf.version} + + + org.apache.cxf + cxf-rt-ws-policy + ${cxf.version} + + + org.apache.cxf + cxf-rt-wsdl + ${cxf.version} + + junit junit - 4.12 + 4.13 test org.mockito mockito-core - 3.1.0 + 3.2.4 test @@ -120,7 +171,7 @@ com.carrotsearch.randomizedtesting randomizedtesting-runner - 2.7.4 + 2.7.6 test @@ -131,6 +182,43 @@ + + + alfresco-public + https://artifacts.alfresco.com/nexus/content/groups/public + + true + + + false + + + + alfresco-public-snapshots + https://artifacts.alfresco.com/nexus/content/groups/public-snapshots + + false + + + true + + + + central + Central Repository + https://repo.maven.apache.org/maven2 + default + + false + + + + maven-restlet + Public online Restlet repository + http://maven.restlet.talend.com + + + alfresco-solr diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableMLTextFieldType.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableMLTextFieldType.java index 03d31c672..89d171642 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableMLTextFieldType.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableMLTextFieldType.java @@ -39,12 +39,9 @@ import org.springframework.extensions.surf.util.I18NUtil; /** * @author Andy - * */ public class AlfrescoCollatableMLTextFieldType extends StrField { - - /* (non-Javadoc) * @see org.apache.solr.schema.StrField#getSortField(org.apache.solr.schema.SchemaField, boolean) */ @@ -75,7 +72,6 @@ public class AlfrescoCollatableMLTextFieldType extends StrField } - public static class MLTextSortFieldComparatorSource extends FieldComparatorSource { @@ -101,16 +97,20 @@ public class AlfrescoCollatableMLTextFieldType extends StrField private final String[] values; private BinaryDocValues docTerms; - - private Bits docsWithField; + + /** + * An array of flags - one for each document in the segment. Each bit is set to true if the document has the + * field or false otherwise. If this is set to null then all docs in the segment have the field. + */ + Bits docsWithField; private final String field; - final Collator collator; + Collator collator; - private String bottom; - - private String top; + String bottom; + + String top; Locale collatorLocale; @@ -138,7 +138,7 @@ public class AlfrescoCollatableMLTextFieldType extends StrField { final String comparableString = findBestValue(doc, docTerms.get(doc)); return compareValues(bottom, comparableString); - + } public void copy(int slot, int doc) @@ -153,13 +153,14 @@ public class AlfrescoCollatableMLTextFieldType extends StrField private String findBestValue(int doc, BytesRef term) { - if (term.length == 0 && docsWithField.get(doc) == false) { + if (term.length == 0 && docsWithField != null && docsWithField.get(doc) == false) + { return null; } - + String withLocale = term.utf8ToString(); - - // split strin into MLText object + + // split string into MLText object if (withLocale == null) { return withLocale; @@ -231,14 +232,15 @@ public class AlfrescoCollatableMLTextFieldType extends StrField { docTerms = DocValues.getBinary(context.reader(), field); docsWithField = DocValues.getDocsWithField(context.reader(), field); - if (docsWithField instanceof Bits.MatchAllBits) { - docsWithField = null; + if (docsWithField instanceof Bits.MatchAllBits) + { + docsWithField = null; } return this; } - + @Override - public int compareValues(String val1, String val2) + public int compareValues(String val1, String val2) { if (val1 == null) { @@ -254,9 +256,10 @@ public class AlfrescoCollatableMLTextFieldType extends StrField } return collator.compare(val1, val2); } - - @Override - public void setScorer(Scorer scorer) {} - } + @Override + public void setScorer(Scorer scorer) + { + } + } } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableTextFieldType.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableTextFieldType.java index d3d54cd63..604dbcc27 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableTextFieldType.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/AlfrescoCollatableTextFieldType.java @@ -104,16 +104,20 @@ public class AlfrescoCollatableTextFieldType extends StrField private final String[] values; private BinaryDocValues docTerms; - - private Bits docsWithField; + + /** + * An array of flags - one for each document in the segment. Each bit is set to true if the document has the + * field or false otherwise. If this is set to null then all docs in the segment have the field. + */ + Bits docsWithField; private final String field; - final Collator collator; + Collator collator; - private String bottom; + String bottom; - private String top; + String top; Locale collatorLocale; @@ -141,7 +145,6 @@ public class AlfrescoCollatableTextFieldType extends StrField { final String comparableString = findBestValue(doc, docTerms.get(doc)); return compareValues(bottom, comparableString); - } public void copy(int slot, int doc) @@ -156,7 +159,8 @@ public class AlfrescoCollatableTextFieldType extends StrField private String findBestValue(int doc, BytesRef term) { - if (term.length == 0 && docsWithField.get(doc) == false) { + if (term.length == 0 && docsWithField != null && docsWithField.get(doc) == false) + { return null; } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/HandlerReportHelper.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/HandlerReportHelper.java index a3c2b069d..72c3e27c0 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/HandlerReportHelper.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/HandlerReportHelper.java @@ -145,9 +145,6 @@ class HandlerReportHelper return payload; } - /** - * Builds Tracker report - */ static NamedList buildTrackerReport(TrackerRegistry trackerRegistry, InformationServer srv, String coreName, Long fromTx, Long toTx, Long fromAclTx, Long toAclTx, Long fromTime, Long toTime) throws JSONException { @@ -157,7 +154,6 @@ class HandlerReportHelper AclTracker aclTracker = trackerRegistry.getTrackerForCore(coreName, AclTracker.class); IndexHealthReport aclReport = aclTracker.checkIndex(toTx, toAclTx, fromTime, toTime); NamedList ihr = new SimpleOrderedMap<>(); - ihr.add("Alfresco version", aclTracker.getAlfrescoVersion()); ihr.add("DB acl transaction count", aclReport.getDbAclTransactionCount()); ihr.add("Count of duplicated acl transactions in the index", aclReport.getDuplicatedAclTxInIndex() .cardinality()); @@ -277,12 +273,6 @@ class HandlerReportHelper Date end = new Date(now.getTime() + remainingTxTimeMillis); Duration remainingTx = new Duration(now, end); - long remainingChangeSetTimeMillis = 0; - - now = new Date(); - end = new Date(now.getTime() + remainingChangeSetTimeMillis); - Duration remainingChangeSet = new Duration(now, end); - NamedList ftsSummary = new SimpleOrderedMap<>(); long remainingContentTimeMillis = 0; srv.addFTSStatusCounts(ftsSummary); diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java index 189a5cf8f..182067271 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/SolrInformationServer.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2015 Alfresco Software Limited. + * Copyright (C) 2019 Alfresco Software Limited. * * This file is part of Alfresco * @@ -81,6 +81,7 @@ import java.util.Map.Entry; import java.util.concurrent.locks.ReentrantReadWriteLock; import java.util.regex.Matcher; import java.util.regex.Pattern; +import java.util.zip.GZIPInputStream; import com.carrotsearch.hppc.IntArrayList; @@ -134,6 +135,7 @@ import org.apache.lucene.analysis.Analyzer; import org.apache.lucene.analysis.TokenStream; import org.apache.lucene.analysis.tokenattributes.CharTermAttribute; import org.apache.lucene.document.Document; +import org.apache.lucene.document.LongPoint; import org.apache.lucene.index.IndexCommit; import org.apache.lucene.index.IndexableField; import org.apache.lucene.index.LeafReaderContext; @@ -164,6 +166,9 @@ import org.apache.solr.schema.SchemaField; import org.apache.solr.search.DelegatingCollector; import org.apache.solr.search.DocIterator; import org.apache.solr.search.DocList; +import org.apache.solr.search.DocSet; +import org.apache.solr.search.QueryCommand; +import org.apache.solr.search.QueryResult; import org.apache.solr.search.QueryWrapperFilter; import org.apache.solr.search.SolrIndexSearcher; import org.apache.solr.update.AddUpdateCommand; @@ -178,14 +183,14 @@ import org.springframework.extensions.surf.util.I18NUtil; import org.springframework.util.FileCopyUtils; /** - * This is the Solr4 implementation of the information server (index). + * This is the Apache Solr implementation of the information server (index). + * * @author Ahmed Owian * @since 5.0 */ public class SolrInformationServer implements InformationServer { private final static Log LOGGER = new Log(SolrInformationServer.class); - private final static long TWO_MINUTES = 120000; private static final String NO_SITE = "_REPOSITORY_"; private static final String SHARED_FILES = "_SHARED_FILES_"; @@ -200,23 +205,23 @@ public class SolrInformationServer implements InformationServer public static final String AND = " AND "; public static final String OR = " OR "; - public static final String REQUEST_HANDLER_ALFRESCO_FULL_TEXT_SEARCH = "/afts"; - public static final String REQUEST_HANDLER_NATIVE = "/native"; - public static final String REQUEST_HANDLER_ALFRESCO = "/alfresco"; - public static final String REQUEST_HANDLER_SELECT = "/select"; - public static final String REQUEST_HANDLER_GET = "/get"; - public static final String RESPONSE_DEFAULT_IDS = "response"; - public static final String RESPONSE_DEFAULT_ID = "doc"; + //public static final String REQUEST_HANDLER_ALFRESCO_FULL_TEXT_SEARCH = "/afts"; + private static final String REQUEST_HANDLER_NATIVE = "/native"; + //public static final String REQUEST_HANDLER_ALFRESCO = "/alfresco"; + //public static final String REQUEST_HANDLER_SELECT = "/select"; + static final String REQUEST_HANDLER_GET = "/get"; + private static final String RESPONSE_DEFAULT_IDS = "response"; + static final String RESPONSE_DEFAULT_ID = "doc"; - public static final String PREFIX_ERROR = "ERROR-"; + static final String PREFIX_ERROR = "ERROR-"; public static final String DOC_TYPE_NODE = "Node"; - public static final String DOC_TYPE_UNINDEXED_NODE = "UnindexedNode"; - public static final String DOC_TYPE_ERROR_NODE = "ErrorNode"; + private static final String DOC_TYPE_UNINDEXED_NODE = "UnindexedNode"; + private static final String DOC_TYPE_ERROR_NODE = "ErrorNode"; public static final String DOC_TYPE_ACL = "Acl"; public static final String DOC_TYPE_TX = "Tx"; public static final String DOC_TYPE_ACL_TX = "AclTx"; - public static final String DOC_TYPE_STATE = "State"; + private static final String DOC_TYPE_STATE = "State"; public static final String SOLR_HOST = "solr.host"; public static final String SOLR_PORT = "solr.port"; @@ -235,7 +240,6 @@ public class SolrInformationServer implements InformationServer private final TrackerStats trackerStats = new TrackerStats(this); private final AlfrescoSolrDataModel dataModel; private final SolrContentStore solrContentStore; - private final String alfrescoVersion; private final boolean transformContent; private final boolean recordUnindexedNodes; private final long lag; @@ -271,7 +275,7 @@ public class SolrInformationServer implements InformationServer protected enum FTSStatus {New, Dirty, Clean} - class DocListCollector implements Collector, LeafCollector + static class DocListCollector implements Collector, LeafCollector { private IntArrayList docs = new IntArrayList(); private int docBase; @@ -303,7 +307,7 @@ public class SolrInformationServer implements InformationServer } } - class TxnCacheFilter extends DelegatingCollector + static class TxnCacheFilter extends DelegatingCollector { private NumericDocValues currentLongs; private Map txnLRU; @@ -330,7 +334,7 @@ public class SolrInformationServer implements InformationServer } } - class TxnCollector extends DelegatingCollector + static class TxnCollector extends DelegatingCollector { private NumericDocValues currentLongs; private long txnFloor; @@ -372,7 +376,7 @@ public class SolrInformationServer implements InformationServer } } - class LRU extends LinkedHashMap + static class LRU extends LinkedHashMap { private int maxSize; @@ -395,7 +399,7 @@ public class SolrInformationServer implements InformationServer boolean isDefinitionExists(QName qName); } - abstract class TransactionInfoReporter + static abstract class TransactionInfoReporter { protected final IndexHealthReport report; @@ -428,7 +432,6 @@ public class SolrInformationServer implements InformationServer this.solrContentStore = solrContentStore; Properties p = core.getResourceLoader().getCoreProperties(); - alfrescoVersion = p.getProperty("alfresco.version", "Unknown"); transformContent = Boolean.parseBoolean(p.getProperty("alfresco.index.transformContent", "true")); recordUnindexedNodes = Boolean.parseBoolean(p.getProperty("alfresco.recordUnindexedNodes", "true")); lag = Integer.parseInt(p.getProperty("alfresco.lag", "1000")); @@ -505,11 +508,6 @@ public class SolrInformationServer implements InformationServer } } - public String getAlfrescoVersion() - { - return this.alfrescoVersion; - } - @Override public void afterInitModels() { @@ -1547,7 +1545,7 @@ public class SolrInformationServer implements InformationServer StringPropertyValue pValue = (StringPropertyValue) properties.get(ContentModel.PROP_IS_INDEXED); if (pValue != null) { - boolean isIndexed = Boolean.valueOf(pValue.getValue()); + boolean isIndexed = Boolean.parseBoolean(pValue.getValue()); if (!isIndexed) { LOGGER.debug("Clearing unindexed"); @@ -1902,7 +1900,7 @@ public class SolrInformationServer implements InformationServer StringPropertyValue pValue = (StringPropertyValue) properties.get(ContentModel.PROP_IS_INDEXED); if (pValue != null) { - boolean isIndexed = Boolean.valueOf(pValue.getValue()); + boolean isIndexed = Boolean.parseBoolean(pValue.getValue()); if (!isIndexed) { LOGGER.debug("Clearing unindexed"); @@ -1958,7 +1956,7 @@ public class SolrInformationServer implements InformationServer } } - private void addToNewDocAndCache(NodeMetaData nodeMetaData, SolrInputDocument newDoc) throws IOException + private void addToNewDocAndCache(NodeMetaData nodeMetaData, SolrInputDocument newDoc) { addFieldsToDoc(nodeMetaData, newDoc); SolrInputDocument cachedDoc = null; @@ -2197,16 +2195,24 @@ public class SolrInformationServer implements InformationServer } } } - + private void deleteErrorNode(UpdateRequestProcessor processor, SolrQueryRequest request, Node node) throws IOException { + String errorDocId = PREFIX_ERROR + node.getId(); - DeleteUpdateCommand delErrorDocCmd = new DeleteUpdateCommand(request); - delErrorDocCmd.setId(errorDocId); - processor.processDelete(delErrorDocCmd); + + // Try finding the node before performing removal operation + DocSet docSet = request.getSearcher().getDocSet(new TermQuery(new Term(FIELD_SOLR4_ID, errorDocId))); + + if (docSet.size() > 0) + { + DeleteUpdateCommand delErrorDocCmd = new DeleteUpdateCommand(request); + delErrorDocCmd.setId(errorDocId); + processor.processDelete(delErrorDocCmd); + } + } - private void deleteNode(UpdateRequestProcessor processor, SolrQueryRequest request, Node node) throws IOException { LOGGER.debug("Deleting node {}", node.getId()); @@ -2215,12 +2221,20 @@ public class SolrInformationServer implements InformationServer // MNT-13767 fix, remove by node DBID. deleteNode(processor, request, node.getId()); } - + private void deleteNode(UpdateRequestProcessor processor, SolrQueryRequest request, long dbid) throws IOException { - DeleteUpdateCommand delDocCmd = new DeleteUpdateCommand(request); - delDocCmd.setQuery(FIELD_DBID + ":" + dbid); - processor.processDelete(delDocCmd); + + // Try finding the node before performing removal operation + DocSet docSet = request.getSearcher().getDocSet(LongPoint.newExactQuery(FIELD_DBID, dbid)); + + if (docSet.size() > 0) + { + DeleteUpdateCommand delDocCmd = new DeleteUpdateCommand(request); + delDocCmd.setQuery(FIELD_DBID + ":" + dbid); + processor.processDelete(delDocCmd); + } + } private boolean isContentIndexedForNode(Map properties) @@ -2232,7 +2246,7 @@ public class SolrInformationServer implements InformationServer .get(ContentModel.PROP_IS_CONTENT_INDEXED); if (pValue != null) { - boolean isIndexed = Boolean.valueOf(pValue.getValue()); + boolean isIndexed = Boolean.parseBoolean(pValue.getValue()); if (!isIndexed) { isContentIndexed = false; @@ -2374,13 +2388,19 @@ public class SolrInformationServer implements InformationServer String transformationStatusFieldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, AlfrescoSolrDataModel.ContentFieldType.TRANSFORMATION_STATUS); - newDoc.addField(transformationStatusFieldName, cachedDoc.getFieldValue(transformationStatusFieldName)); + if (transformationStatusFieldName != null){ + newDoc.addField(transformationStatusFieldName, cachedDoc.getFieldValue(transformationStatusFieldName)); + } String transformationExceptionFieldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, AlfrescoSolrDataModel.ContentFieldType.TRANSFORMATION_EXCEPTION); - newDoc.addField(transformationExceptionFieldName, cachedDoc.getFieldValue(transformationExceptionFieldName)); - String transformationTimeFieldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, + if (transformationExceptionFieldName != null){ + newDoc.addField(transformationExceptionFieldName, cachedDoc.getFieldValue(transformationExceptionFieldName)); + } + String transformationTimeFieldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, AlfrescoSolrDataModel.ContentFieldType.TRANSFORMATION_TIME); - newDoc.addField(transformationTimeFieldName, cachedDoc.getFieldValue(transformationTimeFieldName)); + if (transformationTimeFieldName != null){ + newDoc.addField(transformationTimeFieldName, cachedDoc.getFieldValue(transformationTimeFieldName)); + } // Gets the new content docid and compares to that of the cachedDoc to mark the content as clean/dirty String fldName = getSolrFieldNameForContentPropertyMetadata(propertyQName, @@ -2393,7 +2413,7 @@ public class SolrInformationServer implements InformationServer if(cachedDoc.getFieldValue(fldName) != null) { - long cachedDocContentDocid = Long.valueOf(String.valueOf(cachedDoc.getFieldValue(fldName))); + long cachedDocContentDocid = Long.parseLong(String.valueOf(cachedDoc.getFieldValue(fldName))); long currentContentDocid = contentPropertyValue.getId(); // If we have used out of date content we mark it as dirty // Otherwise we leave it alone - it could already be marked as dirty/New and require an update @@ -2479,8 +2499,12 @@ public class SolrInformationServer implements InformationServer response); addContentPropertyMetadata(doc, propertyQName, AlfrescoSolrDataModel.ContentFieldType.TRANSFORMATION_TIME, response); - + InputStream ris = response.getContent(); + if (Objects.equals(response.getContentEncoding(), "gzip")) + { + ris = new GZIPInputStream(ris); + } String textContent = ""; try { @@ -3584,7 +3608,7 @@ public class SolrInformationServer implements InformationServer field, 1); // Min count of 1 ensures that the id returned is in the index for (Map.Entry idCount : idCounts) { - long idInIndex = Long.valueOf(idCount.getKey()); + long idInIndex = Long.parseLong(idCount.getKey()); // Only looks at facet values that fit the query if (batchStartId <= idInIndex && idInIndex <= batchEndId) diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/content/SolrContentStore.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/content/SolrContentStore.java index a11f3f4ba..1a3694a11 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/content/SolrContentStore.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/content/SolrContentStore.java @@ -43,12 +43,14 @@ import java.io.OutputStream; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.nio.file.StandardCopyOption; import java.text.SimpleDateFormat; import java.util.Date; import java.util.List; import java.util.Locale; import java.util.Map; import java.util.function.Predicate; +import java.util.stream.Stream; import java.util.zip.GZIPInputStream; import java.util.zip.GZIPOutputStream; @@ -236,7 +238,8 @@ public final class SolrContentStore implements Closeable, AccessMode { try { - return Files.lines(Paths.get(root, ".version")) + return Files.readAllLines(Paths.get(root, ".version")) + .stream() .map(Long::parseLong) .findFirst() .orElse(NO_VERSION_AVAILABLE); @@ -250,18 +253,29 @@ public final class SolrContentStore implements Closeable, AccessMode @Override public void setLastCommittedVersion(long version) { + + File tmpFile = new File(root, ".version-" + new SimpleDateFormat(SnapShooter.DATE_FMT, Locale.ROOT).format(new Date())); try { - File tmpFile = new File(root, ".version-" + new SimpleDateFormat(SnapShooter.DATE_FMT, Locale.ROOT).format(new Date())); FileWriter wr = new FileWriter(tmpFile); wr.write(Long.toString(version)); wr.close(); - tmpFile.renameTo(new File(root, ".version")); + // file.renameTo(..) does not work on windows. Use Files.move instead. + Files.move(tmpFile.toPath(), new File(root, ".version").toPath(), StandardCopyOption.ATOMIC_MOVE); + } catch (IOException exception) { logger.error("Unable to persist the last committed content store version {}. See the stacktrace below for furtger details.", version, exception); + try + { + Files.delete(tmpFile.toPath()); + } + catch (IOException e) + { + logger.error("Unable to delete tmp contentstore version file {}.", version); + } } } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoIndexFetcher.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoIndexFetcher.java index ee1666778..bb36889e7 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoIndexFetcher.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoIndexFetcher.java @@ -34,40 +34,10 @@ */ package org.alfresco.solr.handler; -import static java.util.List.of; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.ALIAS; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CHECKSUM; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_CONTENT_STORE_FILES; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_DETAILS; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_GET_FILE; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_GET_FILE_LIST; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_INDEX_VERSION; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.COMMAND; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.COMPRESSION; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONF_FILES; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONF_FILE_SHORT; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_FILES; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_FILE_LIST; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_VERSION; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.EXTERNAL; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FILE; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FILE_STREAM; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FileInfo; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.GENERATION; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.INTERNAL; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.MASTER_URL; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.NO_INDEX_REPLICATION_REQUIRED; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.OFFSET; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.SIZE; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.TLOG_FILE; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.TLOG_FILES; -import static org.alfresco.solr.handler.AlfrescoReplicationHandler.getCheckSum; -import static org.apache.solr.common.params.CommonParams.JAVABIN; -import static org.apache.solr.common.params.CommonParams.NAME; - import com.google.common.base.Strings; import com.google.common.collect.Lists; import org.alfresco.solr.content.SolrContentStore; +import org.apache.commons.io.FilenameUtils; import org.apache.http.client.HttpClient; import org.apache.lucene.codecs.CodecUtil; import org.apache.lucene.index.IndexCommit; @@ -152,6 +122,37 @@ import java.util.zip.Adler32; import java.util.zip.Checksum; import java.util.zip.InflaterInputStream; +import static java.util.List.of; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.ALIAS; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CHECKSUM; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_CONTENT_STORE_FILES; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_DETAILS; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_GET_FILE; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_GET_FILE_LIST; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CMD_INDEX_VERSION; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.COMMAND; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.COMPRESSION; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONF_FILES; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONF_FILE_SHORT; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_FILES; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_FILE_LIST; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.CONTENT_STORE_VERSION; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.EXTERNAL; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FILE; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FILE_STREAM; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.FileInfo; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.GENERATION; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.INTERNAL; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.MASTER_URL; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.NO_INDEX_REPLICATION_REQUIRED; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.OFFSET; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.SIZE; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.TLOG_FILE; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.TLOG_FILES; +import static org.alfresco.solr.handler.AlfrescoReplicationHandler.getCheckSum; +import static org.apache.solr.common.params.CommonParams.JAVABIN; +import static org.apache.solr.common.params.CommonParams.NAME; + /** *

Provides functionality of downloading changed index files as well as config files and a timer for scheduling fetches from the * master.

@@ -188,6 +189,13 @@ class AlfrescoIndexFetcher private final Map confFileInfoCache = new HashMap<>(); private volatile Date replicationStartTimeStamp; private RTimer replicationTimer; + + /** + * The map contains the following fields: + * NAME : String -> file name(with path for contentstore files) + * SIZE : long -> file size + * CHECKSUM : long -> checksum + */ private volatile List> filesToDownload; private volatile List> confFilesToDownload; private volatile List> tlogFilesToDownload; @@ -197,6 +205,7 @@ class AlfrescoIndexFetcher private volatile List> confFilesDownloaded; private volatile List> tlogFilesDownloaded; private volatile List> contentStoreFilesDownloaded; + private volatile Map currentFile; private volatile DirectoryFileFetcher dirFileFetcher; private volatile LocalFsFileFetcher localFileFetcher; @@ -467,6 +476,8 @@ class AlfrescoIndexFetcher Map>> contentStoreMap = (Map>>) response .get(CONTENT_STORE_FILES); + fullContentStoreReplication = false; + if (contentStoreMap != null) { contentStoreFilesToDownload = Collections.synchronizedList(contentStoreMap.get(SolrContentStore.ADDS)); @@ -776,30 +787,37 @@ class AlfrescoIndexFetcher } } - if (contentStoreReplicationNeeded) - { + try { - if (contentStoreFilesToDownload != null) + if (contentStoreReplicationNeeded) { - bytesDownloaded += downloadContentStoreFiles(contentStore.getRootLocation()); + + if (contentStoreFilesToDownload != null) + { + bytesDownloaded += downloadContentStoreFiles(contentStore.getRootLocation()); + } + + if (contentStoreFilesToDelete != null) + { + deleteContentStoreFiles(contentStore.getRootLocation(), contentStoreFilesToDelete); + } + + if (fullContentStoreReplication) + { + cleanUpContentStore(contentStore.getRootLocation()); + } + + contentStore.setLastCommittedVersion(masterContentStoreVersion); + LOG.info("content store has been updated to version: {}", masterContentStoreVersion); } - if (contentStoreFilesToDelete != null) - { - deleteContentStoreFiles(contentStore.getRootLocation(), contentStoreFilesToDelete); - } - - if (fullContentStoreReplication) - { - cleanUpContentStore(contentStore.getRootLocation()); - } - - contentStore.setLastCommittedVersion(masterContentStoreVersion); + } catch (Exception e) { + LOG.error("impossible to complete content store replication {}", e); } final long timeTakenSeconds = getReplicationTimeElapsed(); final Long bytesDownloadedPerSecond = (timeTakenSeconds != 0 ? bytesDownloaded / timeTakenSeconds : - null); + null); LOG.info("Total time taken for download (fullCopy={},bytesDownloaded={}) : {} secs ({} bytes/sec)", isFullCopyNeeded, bytesDownloaded, timeTakenSeconds, bytesDownloadedPerSecond); @@ -1683,37 +1701,30 @@ class AlfrescoIndexFetcher * @param contentStorePath * @throws IOException */ - private void copyTmpContentStoreToContentStore(File tmpContentStoreDir, String contentStorePath) throws IOException + private void copyTmpContentStoreToContentStore(File tmpContentStoreDir, String contentStorePath) throws Exception { String tmpContentStorePath = tmpContentStoreDir.getPath(); - try - { - Files.walk(tmpContentStoreDir.toPath()).forEach(p -> { - File tmpFile = new File(p.toUri()); - if (!tmpFile.isDirectory()) + Files.walk(tmpContentStoreDir.toPath()).forEach(p -> { + File tmpFile = new File(p.toUri()); + if (!tmpFile.isDirectory()) + { + File csFile = new File(p.toString().replace(tmpContentStorePath, contentStorePath)); + try { - File csFile = new File(p.toString().replaceFirst(tmpContentStorePath, contentStorePath)); - try - { - Files.createDirectories(Paths.get(csFile.getParent())); - tmpFile.renameTo(csFile); - } - catch (IOException e) - { - LOG.error("impossible to copy {}", csFile.toString()); - } + Files.createDirectories(Paths.get(csFile.getParent())); + Files.move(tmpFile.toPath(), csFile.toPath(), StandardCopyOption.ATOMIC_MOVE); } - }); - } - catch (IOException e) - { - LOG.error("impossible tmp content store"); - throw e; - } + catch (IOException e) + { + throw new RuntimeException(e); + } + } + }); } + /** * Deletes the files in filesToDelete list from contentStore * @param contentStorePath @@ -1733,16 +1744,23 @@ class AlfrescoIndexFetcher * Deletes from contentstore all the files that has not been updated. * @param contentStorePath */ - private void cleanUpContentStore(String contentStorePath) + private void cleanUpContentStore(String contentStorePath) throws Exception { AtomicInteger fileDeleted = new AtomicInteger(); - Set fileNames = contentStoreFilesToDownload.stream().map(e -> (String) e.get(NAME)) + + // This is the set of the ONLY files that should be in contentStore. + // This set is computed from the information got from master. After a full replication, only the files + // that have been downloaded from master (contentStoreFilesToDownload) should be in contentStore. + // The file paths are translated in the current OS path notation. + Set contentStoreFiles = contentStoreFilesToDownload.stream() + .map(e -> (String) e.get(NAME)) + .map(FilenameUtils::separatorsToSystem) .collect(Collectors.toSet()); try { Files.walk(Paths.get(contentStorePath)).forEach(p -> { File f = new File(p.toUri()); - if (!f.isDirectory() && !fileNames.contains(p.toString().replaceFirst(contentStorePath, ""))) + if (!f.isDirectory() && !contentStoreFiles.contains(p.toString().replace(contentStorePath, ""))) { try { @@ -1756,9 +1774,10 @@ class AlfrescoIndexFetcher } }); } - catch (IOException e) + catch (Exception e) { LOG.error("Impossible to delete unnecessary files. Content store may contains unused contents"); + throw(e); } LOG.info("deleted {} unnecessary files from content store", fileDeleted); diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoReplicationHandler.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoReplicationHandler.java index be01b48b4..b382314e9 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoReplicationHandler.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/handler/AlfrescoReplicationHandler.java @@ -2122,6 +2122,7 @@ public class AlfrescoReplicationHandler extends RequestHandlerBase implements So e.printStackTrace(); } } + } } catch (Exception e) @@ -2169,6 +2170,7 @@ public class AlfrescoReplicationHandler extends RequestHandlerBase implements So if (bytesRead <= 0) { writeNothingAndFlush(); + inputStream.close(); break; } @@ -2185,6 +2187,7 @@ public class AlfrescoReplicationHandler extends RequestHandlerBase implements So fos.flush(); } } + } } } diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/lifecycle/SolrCoreLoadListener.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/lifecycle/SolrCoreLoadListener.java index 66ed18b9e..fc7597a0f 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/lifecycle/SolrCoreLoadListener.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/lifecycle/SolrCoreLoadListener.java @@ -251,7 +251,7 @@ public class SolrCoreLoadListener extends AbstractSolrEventListener MetadataTracker metadataTracker = registerAndSchedule( - new MetadataTracker(true, props, repositoryClient, core.getName(), srv), + new MetadataTracker(true, props, repositoryClient, core.getName(), srv, true), core, props, trackerRegistry, diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/AbstractTracker.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/AbstractTracker.java index d4429f92b..f4d381cd1 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/AbstractTracker.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/AbstractTracker.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2005-2014 Alfresco Software Limited. + * Copyright (C) 2005-2019 Alfresco Software Limited. * * This file is part of Alfresco * @@ -18,6 +18,8 @@ */ package org.alfresco.solr.tracker; +import static java.util.Optional.ofNullable; + import java.lang.invoke.MethodHandles; import java.net.ConnectException; import java.net.SocketTimeoutException; @@ -49,11 +51,10 @@ public abstract class AbstractTracker implements Tracker protected SOLRAPIClient client; InformationServer infoSrv; protected String coreName; - protected StoreRef storeRef; - protected long batchCount; - protected String alfrescoVersion; - protected TrackerStats trackerStats; - protected boolean runPostModelLoadInit = true; + StoreRef storeRef; + long batchCount; + TrackerStats trackerStats; + boolean runPostModelLoadInit = true; private int maxLiveSearchers; private volatile boolean shutdown = false; @@ -63,9 +64,9 @@ public abstract class AbstractTracker implements Tracker protected volatile TrackerState state; protected int shardCount; protected int shardInstance; - protected String shardMethod; + String shardMethod; protected boolean transformContent; - protected String shardTemplate; + String shardTemplate; protected volatile boolean rollback; protected final Type type; @@ -102,12 +103,8 @@ public abstract class AbstractTracker implements Tracker transformContent = Boolean.parseBoolean(p.getProperty("alfresco.index.transformContent", "true")); this.trackerStats = this.infoSrv.getTrackerStats(); - - alfrescoVersion = p.getProperty("alfresco.version", "5.0.0"); this.type = type; - - LOGGER.info("Solr built for Alfresco version: {}", alfrescoVersion); } @@ -183,12 +180,9 @@ public abstract class AbstractTracker implements Tracker if(this.state == null) { - /* - * Set the global state for the tracker here. - */ this.state = getTrackerState(); - LOGGER.debug("##### Setting tracker global state."); - LOGGER.debug("State set: {}", this.state.toString()); + + LOGGER.debug("Global Tracker State set to: {}", this.state.toString()); this.state.setRunning(true); } else @@ -237,12 +231,11 @@ public abstract class AbstractTracker implements Tracker finally { infoSrv.unregisterTrackerThread(); - if(state != null) - { - //During a rollback state is set to null. + ofNullable(state).ifPresent(tstate -> { + // During a rollback state is set to null. state.setRunning(false); state.setCheck(false); - } + }); runLock.release(); } } @@ -284,7 +277,7 @@ public abstract class AbstractTracker implements Tracker /** * Allows time for the scheduled asynchronous tasks to complete */ - protected synchronized void waitForAsynchronous() + synchronized void waitForAsynchronous() { AbstractWorkerRunnable currentRunnable = this.threadHandler.peekHeadReindexWorker(); while (currentRunnable != null) @@ -305,12 +298,12 @@ public abstract class AbstractTracker implements Tracker } } - public int getMaxLiveSearchers() + int getMaxLiveSearchers() { return maxLiveSearchers; } - protected void checkShutdown() + void checkShutdown() { if(shutdown) { @@ -345,20 +338,11 @@ public abstract class AbstractTracker implements Tracker return this.writeLock; } - public Semaphore getRunLock() + Semaphore getRunLock() { return this.runLock; } - /** - * @return Alfresco version Solr was built for - */ - @Override - public String getAlfrescoVersion() - { - return alfrescoVersion; - } - public Properties getProps() { return props; diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/MetadataTracker.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/MetadataTracker.java index 3862ebd30..4821278b4 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/MetadataTracker.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/MetadataTracker.java @@ -40,6 +40,7 @@ import org.alfresco.solr.client.Node.SolrApiNodeStatus; import org.alfresco.solr.client.SOLRAPIClient; import org.alfresco.solr.client.Transaction; import org.alfresco.solr.client.Transactions; +import org.alfresco.util.Pair; import org.apache.commons.codec.EncoderException; import org.json.JSONException; import org.slf4j.Logger; @@ -63,14 +64,86 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker private ConcurrentLinkedQueue nodesToIndex = new ConcurrentLinkedQueue<>(); private ConcurrentLinkedQueue nodesToPurge = new ConcurrentLinkedQueue<>(); private ConcurrentLinkedQueue queriesToReindex = new ConcurrentLinkedQueue<>(); + + /** + * Check if nextTxCommitTimeService is available in the repository. + * This service is used to find the next available transaction commit time from a given time, + * so periods of time where no document updating is happening can be skipped while getting + * pending transactions list. + * + * {@link org.alfresco.solr.client.SOLRAPIClient#GET_NEXT_TX_COMMIT_TIME} + */ + private boolean nextTxCommitTimeServiceAvailable = false; + + /** + * Check if txInteravlCommitTimeService is available in the repository. + * This service returns the minimum and the maximum commit time for transactions in a node id range, + * so method sharding DB_ID_RANGE can skip transactions not relevant for the DB ID range. + * + * {@link org.alfresco.solr.client.SOLRAPIClient#GET_TX_INTERVAL_COMMIT_TIME} + */ + private boolean txIntervalCommitTimeServiceAvailable = false; public MetadataTracker(final boolean isMaster, Properties p, SOLRAPIClient client, String coreName, - InformationServer informationServer) + InformationServer informationServer) + { + this(isMaster, p, client, coreName, informationServer, false); + } + + /** + * MetadataTracker constructor + * + * @param isMaster is true if SOLR instance is master, false otherwise + * @param p includes SOLR core properties (from environment variables and properties file) + * @param client Alfresco Repository http client + * @param coreName Name of the SOLR Core (alfresco, archive) + * @param informationServer SOLR Information Server + * @param checkRepoServicesAvailability is true if Repo Services availability needs to be checked + */ + public MetadataTracker(final boolean isMaster, Properties p, SOLRAPIClient client, String coreName, + InformationServer informationServer, boolean checkRepoServicesAvailability) { super(isMaster, p, client, coreName, informationServer, Tracker.Type.METADATA); transactionDocsBatchSize = Integer.parseInt(p.getProperty("alfresco.transactionDocsBatchSize", "100")); nodeBatchSize = Integer.parseInt(p.getProperty("alfresco.nodeBatchSize", "10")); threadHandler = new ThreadHandler(p, coreName, "MetadataTracker"); + + // In order to apply performance optimizations, checking the availability of Repo Web Scripts is required. + // As these services are available from ACS 6.2 + if (checkRepoServicesAvailability) + { + // Try invoking getNextTxCommitTime service + try + { + client.getNextTxCommitTime(coreName, 0l); + nextTxCommitTimeServiceAvailable = true; + } + catch (NoSuchMethodException e) + { + log.warn("nextTxCommitTimeService is not available. Upgrade your ACS Repository version in order to use this feature: {} ", e.getMessage()); + } + catch (Exception e) + { + log.error("Checking nextTxCommitTimeService failed.", e); + } + + // Try invoking txIntervalCommitTime service + try + { + client.getTxIntervalCommitTime(coreName, 0l, 0l); + txIntervalCommitTimeServiceAvailable = true; + } + catch (NoSuchMethodException e) + { + log.warn("txIntervalCommitTimeServiceAvailable is not available. If you are using DB_ID_RANGE shard method, " + + "upgrade your ACS Repository version in order to use this feature: {} ", e.getMessage()); + } + catch (Exception e) + { + log.error("Checking txIntervalCommitTimeServiceAvailable failed.", e); + } + } + } MetadataTracker() @@ -518,7 +591,7 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker } protected Transactions getSomeTransactions(BoundedDeque txnsFound, Long fromCommitTime, long timeStep, - int maxResults, long endTime) throws AuthenticationException, IOException, JSONException, EncoderException + int maxResults, long endTime) throws AuthenticationException, IOException, JSONException, EncoderException, NoSuchMethodException { long actualTimeStep = timeStep; @@ -546,6 +619,17 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker { transactions = client.getTransactions(startTime, null, startTime + actualTimeStep, null, maxResults, shardstate); startTime += actualTimeStep; + + // If no transactions are found, advance the time window to the next available transaction commit time + if (nextTxCommitTimeServiceAvailable && transactions.getTransactions().size() == 0) + { + Long nextTxCommitTime = client.getNextTxCommitTime(coreName, startTime); + if (nextTxCommitTime != -1) + { + log.info("Advancing transactions from {} to {}", startTime, nextTxCommitTime); + transactions = client.getTransactions(nextTxCommitTime, null, nextTxCommitTime + actualTimeStep, null, maxResults, shardstate); + } + } } while (((transactions.getTransactions().size() == 0) && (startTime < endTime)) || ((transactions.getTransactions().size() > 0) && alreadyFoundTransactions(txnsFound, transactions))); @@ -605,9 +689,46 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker * */ - Long fromCommitTime = getTxFromCommitTime(txnsFound, state.getLastGoodTxCommitTimeInIndex()); + Long fromCommitTime = getTxFromCommitTime(txnsFound, + state.getLastIndexedTxCommitTime() == 0 ? state.getLastGoodTxCommitTimeInIndex() : state.getLastIndexedTxCommitTime()); log.debug("#### Check txnsFound : " + txnsFound.size()); log.debug("======= fromCommitTime: " + fromCommitTime); + + // When using DB_ID_RANGE, fromCommitTime cannot be before the commit time of the first transaction + // for the DB_ID_RANGE to be indexed and commit time of the last transaction cannot be lower than fromCommitTime. + // When there isn't nodes in that range, -1 is returned as commit times + if (docRouter instanceof DBIDRangeRouter && txIntervalCommitTimeServiceAvailable) + { + + DBIDRangeRouter dbIdRangeRouter = (DBIDRangeRouter) docRouter; + Pair commitTimes = client.getTxIntervalCommitTime(coreName, + dbIdRangeRouter.getStartRange(), dbIdRangeRouter.getEndRange()); + Long shardMinCommitTime = commitTimes.getFirst(); + Long shardMaxCommitTime = commitTimes.getSecond(); + + // Node Range it's not still available in repository + if (shardMinCommitTime == -1) + { + log.debug("#### [DB_ID_RANGE] No nodes in range [" + dbIdRangeRouter.getStartRange() + "-" + + dbIdRangeRouter.getEndRange() + "] " + + "exist in the repository. Skipping metadata tracking."); + return; + } + if (fromCommitTime > shardMaxCommitTime) + { + log.debug("#### [DB_ID_RANGE] Last commit time is greater that max commit time in in range [" + + dbIdRangeRouter.getStartRange() + "-" + dbIdRangeRouter.getEndRange() + "]. " + + "Skipping metadata tracking."); + return; + } + // Initial commit time for Node Range is greater than calculated from commit time + if (fromCommitTime < shardMinCommitTime) + { + log.debug("#### [DB_ID_RANGE] SKIPPING TRANSACTIONS FROM " + fromCommitTime + " TO " + + shardMinCommitTime); + fromCommitTime = shardMinCommitTime; + } + } log.debug("#### Get txn from commit time: " + fromCommitTime); transactions = getSomeTransactions(txnsFound, fromCommitTime, TIME_STEP_1_HR_IN_MS, 2000, @@ -964,7 +1085,7 @@ public class MetadataTracker extends CoreStatePublisher implements Tracker } public IndexHealthReport checkIndex(Long toTx, Long toAclTx, Long fromTime, Long toTime) - throws IOException, AuthenticationException, JSONException, EncoderException + throws IOException, AuthenticationException, JSONException, EncoderException, NoSuchMethodException { // DB TX Count long firstTransactionCommitTime = 0; diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/SlaveCoreStatePublisher.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/SlaveCoreStatePublisher.java index 19a93631c..71750c53f 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/SlaveCoreStatePublisher.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/SlaveCoreStatePublisher.java @@ -1,3 +1,21 @@ +/* + * Copyright (C) 2005-2019 Alfresco Software Limited. + * + * This file is part of Alfresco + * + * Alfresco is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Alfresco is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with Alfresco. If not, see . + */ package org.alfresco.solr.tracker; import static org.alfresco.solr.tracker.Tracker.Type.NODE_STATE_PUBLISHER; diff --git a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/Tracker.java b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/Tracker.java index 05fe0766a..976d52c20 100644 --- a/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/Tracker.java +++ b/search-services/alfresco-search/src/main/java/org/alfresco/solr/tracker/Tracker.java @@ -32,8 +32,6 @@ public interface Tracker boolean hasMaintenance() throws Exception; Semaphore getWriteLock(); - - String getAlfrescoVersion(); void setShutdown(boolean shutdown); diff --git a/search-services/alfresco-search/src/main/resources/solr/instance/templates/noRerank/conf/solrconfig.xml b/search-services/alfresco-search/src/main/resources/solr/instance/templates/noRerank/conf/solrconfig.xml index a51492e56..e7bd84f6c 100644 --- a/search-services/alfresco-search/src/main/resources/solr/instance/templates/noRerank/conf/solrconfig.xml +++ b/search-services/alfresco-search/src/main/resources/solr/instance/templates/noRerank/conf/solrconfig.xml @@ -1154,7 +1154,7 @@ https://wiki.apache.org/solr/SolrCloud/ --> - + + 1.5.0 unpack-solr-war @@ -111,7 +111,7 @@ ${project.version} libs ${project.build.directory}/solr-libs - **/jackson-dataformat-smile-*.jar,**/asm-3.3.1.jar,**/jackson-core-asl-*.jar,**/jackson-mapper-asl-*.jar,**/dom4j-1.6.1.jar,**/annotations-1.0.0.jar + **/jackson-dataformat-smile-*.jar,**/asm-3.3.1.jar,**/jackson-core-asl-*.jar,**/jackson-mapper-asl-*.jar,**/dom4j-1.6.1.jar,**/annotations-1.0.0.jar,**/woodstox-core-asl-4.4.1.jar @@ -157,7 +157,9 @@ + + diff --git a/search-services/packaging/src/docker/6.x/docker-compose.yml b/search-services/packaging/src/docker/6.x/docker-compose.yml index 1174827d3..ca4268e79 100644 --- a/search-services/packaging/src/docker/6.x/docker-compose.yml +++ b/search-services/packaging/src/docker/6.x/docker-compose.yml @@ -37,6 +37,10 @@ services: search: image: quay.io/alfresco/search-services:${SEARCH_TAG} environment: + #Replication properties + - REPLICATION_TYPE=master + #- REPLICATION_AFTER=commit,startup- SOLR_ALFRESCO_HOST=alfresco + #- REPLICATION_CONFIG_FILES=schema.xml,stopwords.txt- SOLR_ALFRESCO_PORT=8080 #Solr needs to know how to register itself with Alfresco - SOLR_ALFRESCO_HOST=alfresco - SOLR_ALFRESCO_PORT=8080 @@ -51,6 +55,27 @@ services: - ENABLE_SPELLCHECK=${SEARCH_ENABLE_SPELLCHECK} ports: - 8083:8983 #Browser port + #search_slave: + # image: quay.io/alfresco/search-services:${SEARCH_TAG} + # environment: + # #Replication properties + # - REPLICATION_TYPE=slave + # - REPLICATION_MASTER_HOST=search + # - REPLICATION_MASTER_PORT=8983 + # #- REPLICATION_MASTER_PROTOCOL=http + # #- REPLICATION_CORE_NAME=alfresco + # #- REPLICATION_POLL_INTERVAL=00:00:60 + # #Solr needs to know how to register itself with Alfresco + # - SOLR_ALFRESCO_HOST=alfresco + # - SOLR_ALFRESCO_PORT=8080 + # #Alfresco needs to know how to call solr + # - SOLR_SOLR_HOST=search + # - SOLR_SOLR_PORT=8983 + # #Create the default alfresco and archive cores + # - SOLR_CREATE_ALFRESCO_DEFAULTS=alfresco,archive + # ports: + # - 8084:8983 #Browser port + activemq: image: alfresco/alfresco-activemq:5.15.6 ports: diff --git a/search-services/packaging/src/docker/Dockerfile b/search-services/packaging/src/docker/Dockerfile index 852223ecf..ea60bca76 100644 --- a/search-services/packaging/src/docker/Dockerfile +++ b/search-services/packaging/src/docker/Dockerfile @@ -1,6 +1,6 @@ # Alfresco Search Services ${project.version} Docker Image -FROM alfresco/alfresco-base-java:11.0.1-openjdk-centos-7-6784d76a7b81 +FROM alfresco/alfresco-base-java:11.0.1-openjdk-centos-7-7a6031154417 LABEL creator="Gethin James" maintainer="Alfresco Search Services Team" ENV DIST_DIR /opt/alfresco-search-services diff --git a/search-services/packaging/src/docker/search_config_setup.sh b/search-services/packaging/src/docker/search_config_setup.sh index 40ad069e1..12fc0bf36 100644 --- a/search-services/packaging/src/docker/search_config_setup.sh +++ b/search-services/packaging/src/docker/search_config_setup.sh @@ -1,5 +1,69 @@ #!/bin/bash set -e +# By default its going to deploy "Master" setup configuration with "REPLICATION_TYPE=master". +# Slave replica service can be enabled using "REPLICATION_TYPE=slave" environment value. + +RERANK_TEMPLATE_PATH=$PWD/solrhome/templates/rerank/conf +NORERANK_TEMPLATE_PATH=$PWD/solrhome/templates/noRerank/conf +SOLR_RERANK_CONFIG_FILE=$RERANK_TEMPLATE_PATH/solrconfig.xml +SOLR_NORERANK_CONFIG_FILE=$NORERANK_TEMPLATE_PATH/solrconfig.xml +SOLR_RERANK_CORE_FILE=$RERANK_TEMPLATE_PATH/solrcore.properties +SOLR_NORERANK_CORE_FILE=$NORERANK_TEMPLATE_PATH/solrcore.properties + +if [[ $REPLICATION_TYPE == "master" ]]; then + + findStringMaster='' + + replaceStringMaster="\n\t \n" + + if [[ $REPLICATION_AFTER == "" ]]; then + REPLICATION_AFTER=commit,startup + fi + + if [[ $REPLICATION_CONFIG_FILES == "" ]]; then + REPLICATION_CONFIG_FILES=schema.xml,stopwords.txt + fi + + for i in $(echo $REPLICATION_AFTER | sed "s/,/ /g") + do + replaceStringMaster+="\t\t"$i"<\/str> \n" + done + + if [[ ! -z "$REPLICATION_CONFIG_FILES" ]]; then + replaceStringMaster+="\t\t$REPLICATION_CONFIG_FILES<\/str> \n" + fi + + replaceStringMaster+="\t<\/lst>" + + sed -i "s/$findStringMaster/$findStringMaster$replaceStringMaster/g" $SOLR_RERANK_CONFIG_FILE $SOLR_NORERANK_CONFIG_FILE + sed -i "s/enable.alfresco.tracking=true/enable.alfresco.tracking=true\nenable.master=true\nenable.slave=false/g" $SOLR_RERANK_CORE_FILE $SOLR_NORERANK_CORE_FILE +fi + +if [[ $REPLICATION_TYPE == "slave" ]]; then + + if [[ $REPLICATION_MASTER_PROTOCOL != https ]]; then + REPLICATION_MASTER_PROTOCOL=http + fi + + if [[ $REPLICATION_MASTER_HOST == "" ]]; then + REPLICATION_MASTER_HOST=localhost + fi + + if [[ $REPLICATION_MASTER_PORT == "" ]]; then + REPLICATION_MASTER_PORT=8083 + fi + + if [[ $REPLICATION_POLL_INTERVAL == "" ]]; then + REPLICATION_POLL_INTERVAL=00:00:30 + fi + + sed -i 's//\ + \ + '$REPLICATION_MASTER_PROTOCOL':\/\/'$REPLICATION_MASTER_HOST':'$REPLICATION_MASTER_PORT'\/solr\/${solr.core.name}<\/str>\ + '$REPLICATION_POLL_INTERVAL'<\/str>\ + <\/lst>/g' $SOLR_RERANK_CONFIG_FILE $SOLR_NORERANK_CONFIG_FILE + sed -i "s/enable.alfresco.tracking=true/enable.alfresco.tracking=false\nenable.master=false\nenable.slave=true/g" $SOLR_RERANK_CORE_FILE $SOLR_NORERANK_CORE_FILE +fi SOLR_IN_FILE=$PWD/solr.in.sh @@ -21,8 +85,7 @@ fi # By default Docker Image is using TLS Mutual Authentication (SSL) for communications with Repository # Plain HTTP can be enabled by setting ALFRESCO_SECURE_COMMS to 'none' if [[ "none" == "$ALFRESCO_SECURE_COMMS" ]]; then - sed -i 's/alfresco.secureComms=https/alfresco.secureComms=none/' ${PWD}/solrhome/templates/rerank/conf/solrcore.properties - sed -i 's/alfresco.secureComms=https/alfresco.secureComms=none/' ${PWD}/solrhome/templates/noRerank/conf/solrcore.properties + sed -i 's/alfresco.secureComms=https/alfresco.secureComms=none/' $SOLR_RERANK_CORE_FILE $SOLR_NORERANK_CORE_FILE # Apply also the setting to existing SOLR cores property files when existing if [[ -f ${PWD}/solrhome/alfresco/conf/solrcore.properties ]]; then sed -i 's/alfresco.secureComms=https/alfresco.secureComms=none/' ${PWD}/solrhome/alfresco/conf/solrcore.properties diff --git a/search-services/packaging/src/main/resources/licenses/3rd-party/Apache-like-XPP.txt b/search-services/packaging/src/main/resources/licenses/3rd-party/BSDlike-XPP.txt similarity index 89% rename from search-services/packaging/src/main/resources/licenses/3rd-party/Apache-like-XPP.txt rename to search-services/packaging/src/main/resources/licenses/3rd-party/BSDlike-XPP.txt index fb11965fa..e4553dcaf 100644 --- a/search-services/packaging/src/main/resources/licenses/3rd-party/Apache-like-XPP.txt +++ b/search-services/packaging/src/main/resources/licenses/3rd-party/BSDlike-XPP.txt @@ -1,58 +1,58 @@ -LICENSE FOR THE Extreme! Lab PullParser ------------------------------------------------------------------------- - -Copyright © 2002 The Trustees of Indiana University. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are -met: - -1) All redistributions of source code must retain the above - copyright notice, the list of authors in the original source - code, this list of conditions and the disclaimer listed in this - license; - -2) All redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the disclaimer - listed in this license in the documentation and/or other - materials provided with the distribution; - -3) Any documentation included with all redistributions must include - the following acknowledgement: - - "This product includes software developed by the Indiana - University Extreme! Lab. For further information please visit - http://www.extreme.indiana.edu/" - - Alternatively, this acknowledgment may appear in the software - itself, and wherever such third-party acknowledgments normally - appear. - -4) The name "Indiana Univeristy" and "Indiana Univeristy - Extreme! Lab" shall not be used to endorse or promote - products derived from this software without prior written - permission from Indiana University. For written permission, - please contact http://www.extreme.indiana.edu/. - -5) Products derived from this software may not use "Indiana - Univeristy" name nor may "Indiana Univeristy" appear in their name, - without prior written permission of the Indiana University. - -Indiana University provides no reassurances that the source code -provided does not infringe the patent or any other intellectual -property rights of any other entity. Indiana University disclaims any -liability to any recipient for claims brought by any other entity -based on infringement of intellectual property rights or otherwise. - -LICENSEE UNDERSTANDS THAT SOFTWARE IS PROVIDED "AS IS" FOR WHICH -NO WARRANTIES AS TO CAPABILITIES OR ACCURACY ARE MADE. INDIANA -UNIVERSITY GIVES NO WARRANTIES AND MAKES NO REPRESENTATION THAT -SOFTWARE IS FREE OF INFRINGEMENT OF THIRD PARTY PATENT, COPYRIGHT, OR -OTHER PROPRIETARY RIGHTS.  INDIANA UNIVERSITY MAKES NO WARRANTIES THAT -SOFTWARE IS FREE FROM "BUGS", "VIRUSES", "TROJAN HORSES", "TRAP -DOORS", "WORMS", OR OTHER HARMFUL CODE. LICENSEE ASSUMES THE ENTIRE -RISK AS TO THE PERFORMANCE OF SOFTWARE AND/OR ASSOCIATED MATERIALS, -AND TO THE PERFORMANCE AND VALIDITY OF INFORMATION GENERATED USING -SOFTWARE. - +LICENSE FOR THE Extreme! Lab +------------------------------------------------------------------------ + +Copyright © 2003 The Trustees of Indiana University. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +1) All redistributions of source code must retain the above + copyright notice, the list of authors in the original source + code, this list of conditions and the disclaimer listed in this + license; + +2) All redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the disclaimer + listed in this license in the documentation and/or other + materials provided with the distribution; + +3) Any documentation included with all redistributions must include + the following acknowledgement: + + "This product includes software developed by the Indiana + University Extreme! Lab. For further information please visit + http://www.extreme.indiana.edu/" + + Alternatively, this acknowledgment may appear in the software + itself, and wherever such third-party acknowledgments normally + appear. + +4) The name "Indiana University" and "Indiana University + Extreme! Lab" shall not be used to endorse or promote + products derived from this software without prior written + permission from Indiana University. For written permission, + please contact http://www.extreme.indiana.edu/. + +5) Products derived from this software may not use "Indiana + University" name nor may "Indiana University" appear in their name, + without prior written permission of the Indiana University. + +Indiana University provides no reassurances that the source code +provided does not infringe the patent or any other intellectual +property rights of any other entity. Indiana University disclaims any +liability to any recipient for claims brought by any other entity +based on infringement of intellectual property rights or otherwise. + +LICENSEE UNDERSTANDS THAT SOFTWARE IS PROVIDED "AS IS" FOR WHICH +NO WARRANTIES AS TO CAPABILITIES OR ACCURACY ARE MADE. INDIANA +UNIVERSITY GIVES NO WARRANTIES AND MAKES NO REPRESENTATION THAT +SOFTWARE IS FREE OF INFRINGEMENT OF THIRD PARTY PATENT, COPYRIGHT, OR +OTHER PROPRIETARY RIGHTS.  INDIANA UNIVERSITY MAKES NO WARRANTIES THAT +SOFTWARE IS FREE FROM "BUGS", "VIRUSES", "TROJAN HORSES", "TRAP +DOORS", "WORMS", OR OTHER HARMFUL CODE. LICENSEE ASSUMES THE ENTIRE +RISK AS TO THE PERFORMANCE OF SOFTWARE AND/OR ASSOCIATED MATERIALS, +AND TO THE PERFORMANCE AND VALIDITY OF INFORMATION GENERATED USING +SOFTWARE. + diff --git a/search-services/packaging/src/main/resources/licenses/notice.txt b/search-services/packaging/src/main/resources/licenses/notice.txt index 92c61b013..3ff7a566b 100644 --- a/search-services/packaging/src/main/resources/licenses/notice.txt +++ b/search-services/packaging/src/main/resources/licenses/notice.txt @@ -22,8 +22,8 @@ antlr-3.5.2.jar http://www.antlr.org/ jaxen-1.2.0.jar http://www.cafeconleche.org/jaxen/ -=== Apache variant License === -xpp3-1.1.3_8.jar http://www.extreme.indiana.edu/xgws/xsoap/xpp/ +=== BSD variant License === +xpp3-1.1.4c.jar http://www.extreme.indiana.edu/dist/java-repository/xpp3/licenses/LICENSE.txt === JSON === @@ -32,43 +32,42 @@ json-20160212.jar http://code.google.com/p/json-simple/ === Apache 2.0 === xml-resolver-1.2.jar https://github.com/FasterXML/jackson -neethi-3.0.3.jar http://ws.apache.org/commons/neethi/ +neethi-3.1.1.jar http://ws.apache.org/commons/neethi/ commons-logging-1.2.jar http://jakarta.apache.org/commons/ commons-lang3-3.9.jar http://jakarta.apache.org/commons/ mybatis-3.3.0.jar http://www.mybatis.org/ chemistry-opencmis-commons-impl-1.1.0.jar http://chemistry.apache.org/ chemistry-opencmis-commons-api-1.1.0.jar http://chemistry.apache.org/ -xmlschema-core-2.2.1.jar http://ws.apache.org/commons/XmlSchema/ +xmlschema-core-2.2.3.jar http://ws.apache.org/commons/XmlSchema/ HikariCP-java7-2.4.13.jar https://github.com/brettwooldridge/HikariCP -cxf-core-3.0.12.jar https://cxf.apache.org/ -cxf-rt-bindings-soap-3.0.12.jar https://cxf.apache.org/ -cxf-rt-bindings-xml-3.0.12.jar https://cxf.apache.org/ -cxf-rt-databinding-jaxb-3.0.12.jar https://cxf.apache.org/ -cxf-rt-frontend-jaxws-3.0.12.jar https://cxf.apache.org/ -cxf-rt-frontend-simple-3.0.12.jar https://cxf.apache.org/ -cxf-rt-transports-http-3.0.12.jar https://cxf.apache.org/ -cxf-rt-ws-addr-3.0.12.jar https://cxf.apache.org/ -cxf-rt-ws-policy-3.0.12.jar https://cxf.apache.org/ -cxf-rt-wsdl-3.0.12.jar https://cxf.apache.org/ +cxf-core-3.2.5.jar https://cxf.apache.org/ +cxf-rt-bindings-soap-3.2.5.jar https://cxf.apache.org/ +cxf-rt-bindings-xml-3.2.5.jar https://cxf.apache.org/ +cxf-rt-databinding-jaxb-3.2.5.jar https://cxf.apache.org/ +cxf-rt-frontend-jaxws-3.2.5.jar https://cxf.apache.org/ +cxf-rt-frontend-simple-3.2.5.jar https://cxf.apache.org/ +cxf-rt-transports-http-3.2.5.jar https://cxf.apache.org/ +cxf-rt-ws-addr-3.2.5.jar https://cxf.apache.org/ +cxf-rt-ws-policy-3.2.5.jar https://cxf.apache.org/ +cxf-rt-wsdl-3.2.5.jar https://cxf.apache.org/ mybatis-spring-1.2.5.jar http://www.mybatis.org/ chemistry-opencmis-server-support-1.0.0.jar http://chemistry.apache.org/ chemistry-opencmis-server-bindings-1.0.0.jar http://chemistry.apache.org/ -quartz-2.3.1.jar http://quartz-scheduler.org/ -jackson-core-2.10.1.jar https://github.com/FasterXML/jackson -jackson-annotations-2.10.1.jar https://github.com/FasterXML/jackson -jackson-databind-2.10.1.jar https://github.com/FasterXML/jackson +quartz-2.3.2.jar http://quartz-scheduler.org/ +jackson-core-2.10.2.jar https://github.com/FasterXML/jackson +jackson-annotations-2.10.2.jar https://github.com/FasterXML/jackson +jackson-databind-2.10.2.jar https://github.com/FasterXML/jackson commons-httpclient-3.1-HTTPCLIENT-1265.jar http://jakarta.apache.org/commons/ -spring-aop-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-beans-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-context-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-context-support-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-core-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-expression-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-jcl-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-jdbc-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-orm-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-tx-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ -spring-web-5.1.8.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-aop-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-beans-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-context-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-context-support-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-core-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-expression-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-jdbc-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-orm-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-tx-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ +spring-web-5.2.2.RELEASE.jar http://projects.spring.io/spring-framework/ xercesImpl-2.12.0-alfresco-patched-20191004.jar http://xerces.apache.org/xerces2-j guessencoding-1.4.jar http://docs.codehaus.org/display/GUESSENC/ xml-apis-1.4.01.jar https://github.com/FasterXML/jackson @@ -87,6 +86,7 @@ jetty-servlets-9.3.27.v20190418.jar https://www.eclipse.org/jetty/licenses.html jetty-util-9.3.27.v20190418.jar https://www.eclipse.org/jetty/licenses.html jetty-webapp-9.3.27.v20190418.jar https://www.eclipse.org/jetty/licenses.html jetty-xml-9.3.27.v20190418.jar https://www.eclipse.org/jetty/licenses.html +woodstox-core-5.0.3.jar https://github.com/FasterXML/woodstox === CDDL 1.0 === @@ -115,8 +115,8 @@ asm-commons-5.1.jar aspectjrt-1.8.0.jar attributes-binder-1.3.1.jar avatica-core-1.9.0.jar -bcmail-jdk15-1.45.jar -bcprov-jdk15-1.45.jar +bcmail-jdk15on-1.47.jar +bcprov-jdk15on-1.47.jar boilerpipe-1.1.0.jar caffeine-2.4.0.jar calcite-core-1.11.0.jar @@ -131,7 +131,7 @@ commons-compiler-2.7.6.jar commons-compress-1.14.jar commons-configuration-1.6.jar commons-exec-1.3.jar -commons-fileupload-1.3.2.jar +commons-fileupload-1.3.3.jar commons-io-2.5.jar commons-lang-2.6.jar commons-math3-3.4.1.jar @@ -168,28 +168,28 @@ jul-to-slf4j-1.7.7.jar juniversalchardet-1.0.3.jar langdetect-1.1-20120112.jar log4j-1.2.17.jar -lucene-analyzers-common-6.6.5-patched.1.jar -lucene-analyzers-icu-6.6.5-patched.1.jar -lucene-analyzers-kuromoji-6.6.5-patched.1.jar -lucene-analyzers-morfologik-6.6.5-patched.1.jar -lucene-analyzers-phonetic-6.6.5-patched.1.jar -lucene-analyzers-smartcn-6.6.5-patched.1.jar -lucene-analyzers-stempel-6.6.5-patched.1.jar -lucene-backward-codecs-6.6.5-patched.1.jar -lucene-classification-6.6.5-patched.1.jar -lucene-codecs-6.6.5-patched.1.jar -lucene-core-6.6.5-patched.1.jar -lucene-expressions-6.6.5-patched.1.jar -lucene-grouping-6.6.5-patched.1.jar -lucene-highlighter-6.6.5-patched.1.jar -lucene-join-6.6.5-patched.1.jar -lucene-memory-6.6.5-patched.1.jar -lucene-misc-6.6.5-patched.1.jar -lucene-queries-6.6.5-patched.1.jar -lucene-queryparser-6.6.5-patched.1.jar -lucene-sandbox-6.6.5-patched.1.jar -lucene-spatial-extras-6.6.5-patched.1.jar -lucene-suggest-6.6.5-patched.1.jar +lucene-analyzers-common-6.6.5-patched.2.jar +lucene-analyzers-icu-6.6.5-patched.2.jar +lucene-analyzers-kuromoji-6.6.5-patched.2.jar +lucene-analyzers-morfologik-6.6.5-patched.2.jar +lucene-analyzers-phonetic-6.6.5-patched.2.jar +lucene-analyzers-smartcn-6.6.5-patched.2.jar +lucene-analyzers-stempel-6.6.5-patched.2.jar +lucene-backward-codecs-6.6.5-patched.2.jar +lucene-classification-6.6.5-patched.2.jar +lucene-codecs-6.6.5-patched.2.jar +lucene-core-6.6.5-patched.2.jar +lucene-expressions-6.6.5-patched.2.jar +lucene-grouping-6.6.5-patched.2.jar +lucene-highlighter-6.6.5-patched.2.jar +lucene-join-6.6.5-patched.2.jar +lucene-memory-6.6.5-patched.2.jar +lucene-misc-6.6.5-patched.2.jar +lucene-queries-6.6.5-patched.2.jar +lucene-queryparser-6.6.5-patched.2.jar +lucene-sandbox-6.6.5-patched.2.jar +lucene-spatial-extras-6.6.5-patched.2.jar +lucene-suggest-6.6.5-patched.2.jar metadata-extractor-2.9.1.jar metrics-core-3.2.2.jar metrics-ganglia-3.2.2.jar @@ -213,11 +213,11 @@ rome-1.5.1.jar simple-xml-2.7.1.jar slf4j-api-1.7.7.jar slf4j-log4j12-1.7.7.jar -solr-analysis-extras-6.6.5-patched.1.jar -solr-clustering-6.6.5-patched.1.jar -solr-core-6.6.5-patched.1.jar -solr-langid-6.6.5-patched.1.jar -solr-solrj-6.6.5-patched.1.jar +solr-analysis-extras-6.6.5-patched.2.jar +solr-clustering-6.6.5-patched.2.jar +solr-core-6.6.5-patched.2.jar +solr-langid-6.6.5-patched.2.jar +solr-solrj-6.6.5-patched.2.jar spatial4j-0.6.jar start.jar stax2-api-3.1.4.jar @@ -229,7 +229,6 @@ tika-parsers-1.16.jar tika-xmp-1.16.jar vorbis-java-core-0.8.jar vorbis-java-tika-0.8.jar -woodstox-core-asl-4.4.1.jar xmlbeans-2.6.0.jar xmpcore-5.1.2.jar xz-1.6.jar diff --git a/search-services/pom.xml b/search-services/pom.xml index 8af1e3219..9a316f644 100644 --- a/search-services/pom.xml +++ b/search-services/pom.xml @@ -14,7 +14,8 @@ pom Alfresco Solr Search parent - 1.7.29 + 1.7.30 + 3.2.5 alfresco-solrclient-lib