30342: Dev branch for Site performance issues (including rework of AuthorityService.getAuthorities() to use a 'lazy' set and DM indexing rework)
ALF-9899 Huge share site migration, add group to site and user access site related performance issue.
ALF-9208 Performance issue, during load tests /share/page/user/user-sites is showing to be the most expensive.
ALF-9692 Performance: General performance of Alfresco degrades when there are 1000s of sites present
- ancestor-preloading
- hasAuthority
- huge site test
30370: - Save changed to do with adding childAuthorityCache to AuthorityDAOImpl
- Increase aspectsTransactionalCache size as it blows up
30387: Experimental solution to 'cascading reindex' performance problem
- Now only Lucene container documents for a single subtree are reprocessed on addition / removal of a secondary child association
- No need to delete and re-evaluate ALL the paths to all the nodes in the subtree - just the paths within the subtree
- Lucene deltas now store the IDs of ANCESTORs to mask out as well as documents to reindex
- Merge handles deletion of these efficiently
- Node service cycle checks changed from getPaths to recursive cycleCheck method
- Adding a group to 60,000 sites might not require all paths to all sites to be re-evaluated on every change!
30389: Missed files from last checkin
30390: Optimizations / fixes to Alan's test!
30393: Bug fix - wasn't adding new documents into the index!
30397: Fixed a problem with bulk loading trying to bulk load zero parent associations
Also tweaked reindex calls
30399: Correction - don't cascade below containers during path cascading
30400: Another optimization - no need to trigger node bulk loading during path cascading - pass false for the preload flag
30404: Further optimizations
- On creation of a secondary child association, make a decision on whether it is cheaper to cascade reindex the parent or the child, based on the number of parent associations to the child
- Assumes that if there are more than 5 parent associations, it's cheaper to cascade reindex the parent
- Add a new authority to a zone (containing 60,000 authorities) - cascade reindex the authority, not the zone
- Add a group (in 60,000 sites) to a site - cascade reindex the site, not the group
- Caching of child associations already traversed during cascade reindexing
- Site creation time much reduced!
30407: Logic fix: Use 'delete only nodes' behaviour on DM index filtering and merging, now we are managing container deletions separately
30408: Small correction related to last change.
30409: Correction to deletion reindex behaviour (no need to regenerate masked out containers)
- Site CRUD operations now all sub-second with 60,000 sites!
30410: Stop the heartbeat from trying to load and count all site groups
- Too expensive, as we might have 60,000 sites, each with 4 groups
- Now just counts the groups in the default zone (the UI visible ones)
30411: Increased lucene parameters to allow for 'path explosion'
- 9 million lucene documents in my index after creating 60,000 Share sites (most of them probably paths) resulting in sluggish index write performance
- Set lucene.indexer.mergerTargetIndexCount=8 (142 documents in smallest index)
- Increased lucene.indexer.maxDocsForInMemoryMerge, lucene.indexer.maxDocsForInMemoryIndex
30412: Test fixes
30413: Revert 'parent association batch loading' changes (as it was a bad idea and is no longer necessary!)
- Retain a few caching bug fixes however
30416: Moved UserAuthoritySet (lazy load authority set) from PermissionServiceImpl to AuthorityServiceImpl
30418: - Remove 'new' hasAuthority from authorityService so it is back to where we started.
- SiteServiceHugeTest minor changes
30421: Prevent creation of a duplicate root node on updating the root
- Use the ANCESTOR field rather than ISCONTAINER to detect a node document, as the root node is both a container and a node!
30447: Pulled new indexing behaviour into ADMLuceneIndexerImpl and restored old behaviour to AVMLuceneIndexerImpl to restore normal AVM behaviour
30448: - Cache in PermissionServiceImpl cleared if an authority container has an association added or removed
Supports the generateKey method which includes the username
Supports changes in group structures
- Moved logic to do with ROLE_GUEST from PermissionServiceImpl to AuthorityServiceImpl
30465: - Tidy up tests in SiteServiceTestHuge
30532: - Added getContainingAuthoritiesInZone to AuthorityService
- Dave Changed PeopleService.getContainerGroups to only return groups in the DEFAULT zone
- Fixed RM code to use getAuthoritiesForUser method with just the username again.
30558: Build fixes
- Fixed cycleCheck to throw a CyclicChildRelationshipException
- More tidy up of AVM / ADM indexer split
- Properly control when path generation is cascaded (not required on a full reindex or a tracker transaction)
- Support indexing of a 'fake root' parent. Ouch my head hurts!
30588: Build fixes
- StringIndexOutOfBoundsException in NodeMonitor
- Corrections to 'node only' delete behaviour
- Use the PATH field to detect non-leaf nodes (it's the only stored field with which we can recognize the root)
- Moved DOD5015Test.testVitalRecords() to the end - the only way I could work out how to get the full TestCase to run
30600: More build fixes
- Broadcast ALL node deletions to indexer (even those from cascade deletion of primary associations)
- Allows indexer to wipe out all affected documents from the delta even if some have already been flushed under different parents by an intricate DOD unit test!
- Pause FTS in DOD5015Test to prevent intermittent test failures (FTS can temporarily leave deleted documents in the index until it catches up)
- More tidy up of ADMLuceneIndexerImpl
- flushPending optimized and some unnecessary member variables removed
- correction to cascade deletion behaviour (leave behind containers of unaffected secondary references)
- unused MOVE action removed
- further legacy logic moved into AVMLuceneIndexerImpl
30620: More build fixes
- Cope with a node morphing from a 'leaf' to a container during its lifetime
- Container documents now created lazily in index as and when necessary
- Blank out 'nth sibling' field of synthesized paths
- ADMLuceneTest now passes!
- TaggingServiceImplTest also passes - more special treatment for categories
30627: Multi tenancy fixes
30629: Possible build fix - retrying transaction in ReplicationServiceIntegrationTest.tearDown()
30632: Build fix - lazy container generation after a move
30636: Build fix: authority comparisons are case sensitive, even when that authority corresponds to a user (PermissionServiceTest.testPermissionCase())
30638: Run SiteServiceTestHuge form a cmd line
set SITE_CPATH=%TOMCAT_HOME%/lib/*;%TOMCAT_HOME%/endorsed/*;%TOMCAT_HOME%/webapps/alfresco/WEB-INF/lib/*;\
%TOMCAT_HOME%/webapps/alfresco/WEB-INF/classes;%TOMCAT_HOME%/shared/classes;
java -Xmx2048m -XX:MaxPermSize=512M -classpath %SITE_CPATH% org.alfresco.repo.site.SiteServiceTestHuge ...
Usage: -Daction=usersOnly
-Dfrom=<fromSiteId> -Dto=<toSiteId>
-Dfrom=<fromSiteId> -Dto=<toSiteId> -Daction=sites -Drestart=<restartAtSiteId>
-Dfrom=<fromSiteId> -Dto=<toSiteId> -Daction=groups -Drestart=<restartAtSiteId>
30639: Minor changes to commented out command line code for SiteServiceTestHuge
30643: Round of improvements to MySites dashlet relating to huge DB testing:
- 10,000 site database, user is a member of ~2000 sites
- Improvements to site.lib.ftl and related SiteService methods
- To return MySites dashlet for the user, order of magnitude improvement from 7562ms to 618ms in the profiler (now ~350ms in the browser)
30644: Fixed performance regression - too much opening and closing of the delta reader and writer
30661: More reader opening / closing
30668: Performance improvements to Site Finder and My Sites in user profile page.
- faster to bring back lists and site memberships (used by the Site Finder)
- related further improvements to APIs used by this and My Sites on dashboard
30713: Configuration for MySites dashlet maximum list size
30725: Merged V3.4-BUG-FIX to DEV/ALAN/SITE_PERF
30708: ALF-10040: Added missing ReferenceCountingReadOnlyIndexReaderFactory wrapper to IndexInfo.getMainIndexReferenceCountingReadOnlyIndexReader() to make it consistent with IndexInfo.getMainIndexReferenceCountingReadOnlyIndexReader(String, Set<String>, boolean) and allow SingleFieldSelectors to make it through from LeafScorer to the path caches! Affects ALL Lucene queries that run OUTSIDE of a transaction.
30729: Use getAuthoritiesForUser rather than getContainingAuthorities if possible.
SiteServiceTestHuge: command line version
30733: Performance improves to user dashboard relating to User Calendar
- converted web-tier calendar dashlet to Ajax client-side rendering - faster user experience and also less load on the web-tier
- improvements to query from Andy
- maximum sites/list size to query now configurable (default 100 instead of previously 1000)
30743: Restore site CRUD performance from cold caches
- Introduced NodeService.getAllRootNodes(), returning all nodes in a store with the root aspect, backed by a transactional cache and invalidated at key points
- Means indexing doesn't have to load all parent nodes just to check for 'fake roots'
- Site CRUD performance now back to sub-second with 60,000 nodes
30747: Improvement to previous checkin - prevent cross cluster invalidation of every store root when a single store drops out of the cache
30748: User dashboard finally loading within seconds with 60,000 sites, 60 groups, 100 users (thanks mostly to Kev's UI changes)
- post-process IBatis mapped statements with MySQL dialect to apply fetchSize=Integer.MIN_VALUE to all _Limited statements
- Means we can stream first 10,000 site groups without the MySQL JDBC driver reading all 240,000 into memory
- New NodeService getChildAssocs method with a maxResults argument (makes use of the above)
- Perfected getContainingAuthoritiesInZone implementation, adding a cutoff parameter, allowing only the first 1000 site memberships to be returned quickly and caches to be warmed for ACL evaluations
- New cache of first 10,000 groups in APP.SHARE zone
- Cache sizes tuned for 60,000 site scenario
- Site service warms caches on bootstrap
- PreferencesService applies ASPECT_IGNORE_INHERITED_RULES to person node to prevent the rule service trying to crawl the group hierarchy on a preference save
- WorkflowServiceImpl.getPooledTasks only looks in APP.DEFAULT zone (thus avoiding site group noise)
30749: Fix compilation errors
30761: Minor change to SiteServiceTestHuge
30762: Derek code review: Reworked fetchSize specification for select_ChildAssocsOfParent_Limited statement for MySQL
- Now fetchSize stated explicitly in a MySQL specific config file resolved by the HierarchicalResourceLoader
- No need for any Java-based post processing
30763: Build fix: don't add a user into its own authorities (until specifically asked to)
30767: Build fix
- IBatis / MySQL needs a streaming result statement to be run in an isolation transaction (because it doesn't release PreparedStatements until the end)
30771: Backed out previous change which was fundamentally flawed
- Resolved underlying problem which was that the select_ChildAssocsOfParent_Limited SQL string needs to be unique in order to not cause confusion in the prepared statement cache
30772: Backed out previous change which was fundamentally flawed
- Resolved underlying problem which was that the select_ChildAssocsOfParent_Limited SQL string needs to be unique in order to not cause confusion in the prepared statement cache
git-svn-id: https://svn.alfresco.com/repos/alfresco-enterprise/alfresco/HEAD/root@30797 c4b6b30b-aa2e-2d43-bbcb-ca4b014f7261
17385: Minor performance tweaks and code cleanup/consolidation in SiteServiceImpl.
17386: ETHREEOH-3066 Added supporting logging
17387: Minor javadoc updates pertinent to the FreeMarker API chapter updates.
17392: Fix to unreported issue in User Profile where a recent fix incorrectly encoding javascript values as ?html.
- values passed into JS components in FreeMarker template should use ?js_string operator - not the ?html operator.
17399: ETHREEOH-3269, ETHREEOH-3271 - Dropdown markup changed for keyboard access
17400: ETHREEOH-3278 - Navigator is very wide
17401: Concatenate DocLib JavaScript files
git-svn-id: https://svn.alfresco.com/repos/alfresco-enterprise/alfresco/HEAD/root@18120 c4b6b30b-aa2e-2d43-bbcb-ca4b014f7261
17246: ETHREEOH-3208: User profiles for users authenticated by immutable subsystems are now read only
- Introduced MutableAuthenticationService interface, only implemented by Alfresco native authentication service
- Split out those methods from AuthenticationService that mutate the user store and added isAuthenticationMutable()
- Now both Alfresco Explorer and Share user profile / password edit link rendering is conditional on isAuthenticationMutable
- Works with authentication chain containing mixture of internally and externally authenticated users
17247: Fix failing unit tests
- rm-public-services-security-context.xml needed to be brought in line with public-services-security-context.xml (and will forever more!)
17248: ETHREEOH-1593: alfUser cookie value should be base 64 encoded to allow for non-ASCII characters
17253: *RECORD ONLY* ETHREEOH-2885: web.xml must conform to the schema to work on JBoss
git-svn-id: https://svn.alfresco.com/repos/alfresco-enterprise/alfresco/HEAD/root@18098 c4b6b30b-aa2e-2d43-bbcb-ca4b014f7261
17667: Branch for SpringSurf integration - from HEAD r17665
17668: Fix to ensure included scripts files are not loaded from a cached classpath loader.
17670: Part 1 of SpringSurf integration - changes relating to spring-surf-core-1.0.0.CI-SNAPSHOT.jar
17674: Part 2 of SpringSurf integration - changes relating to spring-surf-core-configservice-1.0.0.CI-SNAPSHOT.jar
git-svn-id: https://svn.alfresco.com/repos/alfresco-enterprise/alfresco/HEAD/root@17788 c4b6b30b-aa2e-2d43-bbcb-ca4b014f7261
16662: LDAP sync: improved group association filtering, referential integrity checking, deletion strategy and performance tuning of batch sizes
16648: ETHREEOH-2752: Improved ticket validation fix
- Invalidate user's tickets during person deletion rather than validation or it can mess up chained validation
16647: ETHREEOH-2534: Fixed Sharepoint NTLM authentication
- user details were never getting cached in the session
16579: Small improvement to LDAP error reporting
- Committed errors counted before successes in a logging interval
16515: LDAP sync performance
- Improved full sync strategy - run differential queries to work out required updates/additions and full queries to work out required deletions. Saves updating unchanged nodes.
- Use a TreeSet rather than a HashSet to gather group associations in an attempt to avoid blowing the heap size
16498: More LDAP performance improvements
- Uses thread pool with 4 worker threads and blocking queue to process returned results. The number of worker threads can be controlled by the synchronization.workerThreads property.
- Switched LDAP connection pooling back on again
- Group Associations processsed individually so that errors are collated and we get a better idea of their throughput
- Fixed potential bug. Group membership resolution done with isolated LDAP context to avoid cookies from paging creeping in.
16424: Try switching off LDAP connection pooling to see if it works better with our flaky server.
16414: Further LDAP fault tolerance
- Log causes of group member resolution failures where possible
16413: More fault tolerance for LDAP sync
- Always commit last sync times before overall sync is complete to avoid the 'forgetting' of differential sync information
- DN comparisons should be case insensitive to avoid issues resolving DNs to user and group IDs
16398: Improved monitoring and fault tolerance for LDAP sync
- When the batch is complete a summary of the number of errors and the last error stack trace will be logged at ERROR level
- Each individual error is logged at WARN level and progress information (including % complete) is collated and logged at INFO level after a configurable interval
- In the Enterprise Edition all metrics can be monitored in real time through JMX
- Sanity testing to be performed by Mike!
16319: Merged HEAD to V3.2
16316: ALFCOM-3397: JBoss 5 compatibility fix
- Relative paths used by LDAP subsystem configuration weren't being resolved correctly
- See also https://jira.jboss.org/jira/browse/JBAS-6548 and https://jira.springsource.org/browse/SPR-5120
16272: ETHREEOH-2752: Once more with feeling!
16261: ETHREEOH-2752: Correct exception propagation.
16260: ETHREEOH-2752: Fix ticket validation
- Current ticket was getting forgotten by previous fix
- Person validation in CHECK mode now done AFTER the current user is set, so that the current ticket is remembered
16243: ETHREEOH-2752: Improve ticket validation used by all authentication filters
- Now takes into account whether person actually exists or not
- Tickets for non-nonexistent persons are now considered invalid and cached session information is invalidated
- New BaseAuthenticationFilter superclass for all authentication filters
- Improved fix to ETHREEOH-2839: WebDAV user is cached consistently using a different session attribute from the Web Client
16233: ETHREEOH-2754: Correction to previous checkin.
- relogin for SSO authentication, logout for normal login page
- logout is default
16232: ETHREEOH-2754: Log Out Action outcome passed as a parameter
- relogin for SSO authentication, login for normal login page
- Means the log out link always leads to the correct place, even when the session has expired
- Also lowered ticket validation error logging to DEBUG level to avoid unnecessary noise in the logs from expired sessions
16220: ETHREEOH-2839: Fixed potential ClassCastExceptions when Alfresco accessed via WebDAV and Web Client links in same browser
- WebDAV side no longer directly casts session user to a WebDAVUser
- ContextListener no longer casts session user to web client user
- Web client side will 'promote' session user to a web client User if necessary via AuthenticationHelper
- All authentication filters made to use appropriate AuthenticationHelper methods
16211: ETHREEOH-2835: LDAP sync batches user and group deletions as well as creations
- Also improved logging of sync failures
16197: ETHREEOH-2782: LDAP subsystems now support search-based user DN resolution
- When ldap.authentication.userNameFormat isn't set (now the default) converts a user ID to a DN by running ldap.synchronization.personQuery with an extra condition tacked on the end to find the user by ID
- Structured directories and authentication by attributes not in the DN such as email address now supported
16189: ALFCOM-3283: Prevent errors when user accepts an invite when not logged in
- new isGuest attribute propagated to user object
- header component (used by accept-invite page) needs to avoid calling prefs and site webscripts for guest user
- Conditional stuff in header template changed to use user.isGuest
git-svn-id: https://svn.alfresco.com/repos/alfresco-enterprise/alfresco/HEAD/root@16896 c4b6b30b-aa2e-2d43-bbcb-ca4b014f7261
16092: ETHREEOH-2800: org.alfresco.repo.jscript.People should use AuthenticationService rather than the MutableAuthenticationDAO or otherwise it won't work in an authentication chain.
16094: ETHREEOH-2800: The same for org.alfresco.repo.template.People
git-svn-id: https://svn.alfresco.com/repos/alfresco-enterprise/alfresco/HEAD/root@16095 c4b6b30b-aa2e-2d43-bbcb-ca4b014f7261
Repository JavaScript API now supports getting/setting user account enabled status (admin authority required).
Repository Template API now supports retrieving user account enabled status.
Fixed up line endings on template-services-context.xml.
git-svn-id: https://svn.alfresco.com/repos/alfresco-enterprise/alfresco/HEAD/root@13884 c4b6b30b-aa2e-2d43-bbcb-ca4b014f7261
Basic "people" API for FreeMarker - just getPerson() and getGroup() currently.
Template cache size and delay now configurable in RepositoryTemplateProcessor (like they already are in PresentationTemplateProcessor) - related default cache size tweaks.
git-svn-id: https://svn.alfresco.com/repos/alfresco-enterprise/alfresco/HEAD/root@13788 c4b6b30b-aa2e-2d43-bbcb-ca4b014f7261
