Vfrdtyky

I keep encountering the below error message in the sha256_migration.log. It doesn't restart after failure, however if I restart the artifactory service it begins the SHA256 migration from where it left off until it fails again.

2018-11-13 10:24:35,060 [art-exec-3] [ERROR] (o.a.s.j.m.s.Sha256MigrationJob:78) - Caught unexpected exception during SHA256 Migration job, operation will break.

org.springframework.core.task.TaskRejectedException: Executor [org.artifactory.schedule.ArtifactoryConcurrentExecutor@70a2137a] did not accept task: org.artifactory.schedule.aop.AsyncAdvice$$Lambda$654/1640835804@7dbf55d3

        at org.springframework.core.task.support.TaskExecutorAdapter.submit(TaskExecutorAdapter.java:93)

        at org.springframework.scheduling.concurrent.ConcurrentTaskExecutor.submit(ConcurrentTaskExecutor.java:143)

        at org.artifactory.schedule.aop.AsyncAdvice.submitWorkQueueTask(AsyncAdvice.java:235)

        at org.artifactory.schedule.aop.AsyncAdvice.submit(AsyncAdvice.java:217)

        at org.artifactory.schedule.aop.AsyncAdvice.executeInvocation(AsyncAdvice.java:146)

        at org.artifactory.schedule.aop.AsyncAdvice.invoke(AsyncAdvice.java:124)

        at org.artifactory.schedule.aop.AsyncAdvice.invoke(AsyncAdvice.java:62)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)

        at com.sun.proxy.$Proxy144.updateSha2(Unknown Source)

        at org.artifactory.storage.jobs.migration.sha256.Sha256MigrationJob.migrationLogic(Sha256MigrationJob.java:134)

        at org.artifactory.storage.jobs.migration.MigrationJobBase.migrationLoop(MigrationJobBase.java:106)

        at org.artifactory.storage.jobs.migration.MigrationJobBase.runMigration(MigrationJobBase.java:83)

        at org.artifactory.storage.jobs.migration.MigrationJobBase.onExecute(MigrationJobBase.java:73)

        at org.artifactory.schedule.quartz.QuartzCommand.execute(QuartzCommand.java:48)

        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

        at org.artifactory.concurrent.ArtifactoryRunnable.run(ArtifactoryRunnable.java:30)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.RejectedExecutionException: Task org.artifactory.concurrent.ArtifactoryRunnable@4afb003 rejected from java.util.concurrent.ThreadPoolExecutor@33daf5aa[Running, pool size = 64, active threads = 64, queued tasks = 10000, completed tasks = 120723]

        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)

        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)

        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)

        at org.artifactory.schedule.ArtifactoryConcurrentExecutor.execute(ArtifactoryConcurrentExecutor.java:69)

        at org.springframework.core.task.support.TaskExecutorAdapter.submit(TaskExecutorAdapter.java:88)

        ... 19 common frames omitted

My artifactory.system.properties regarding sha256migration

##SHA2 Migration block

artifactory.sha2.migration.job.enabled=true

artifactory.sha2.migration.job.queue.workers=100

My setup:

• Cloned production instances of Artifactory infrastructure into test instance (save for IP address and DNS records).


• Ensure that DB and filestore configurations (db.properties and binarystore.xml) were updated accordingly on the cloned instances.

Things I've tried without luck:

• I ran the Artifactory GC couple of times.


• Increase the CPU count to 16


• Increase the RAM to 16G


• Ensure that I am running the latest Oracle Java 8u192

What I know:

• It keep running fine for a while until it crashes.


• When I restart artifactory service, the migration resumes and the total artifacts to migrate is lower.


• I cannot keep restarting Artifactory in production to finish the sha256migrationjob, I have over 500k artifacts.

My question:

• Any way method to restart the SHA256MigrationJob without restarting Artifactory?


• Is there a way to find the artifact that it has trouble migrating to SHA256?


• In the stack trace above, I feel the issue is at com.sun.proxy.$Proxy144.updateSha2(Unknown Source).

-- Workaround --

I ended up creating a new VM and installed a clean copy Artifactory 6.5.3 (with latest Oracle Java 8 Server-JRE). In the above issue, I was doing an in-place upgrade, just in a new folder.

I moved the necessary files in the etc to the new VM; such as master.key, binarystore.xml, db.properties and etc. I then executed the bin/installService.sh [user] [group], this creates the /etc/opt/jfrog/artifactory configuration symlink/folder move. My filestore and artifactory database are running on different VMs, thus only Artifactory and it's file configurations needed to be ported.

The new Artifactory 6.5.3 version started up without issues.

The sha256migrationjob is actually running without any problems. Last upgrade run I did, it worked fine without the job dying.

Notes: I did also sanely adjust the configuration values on the queue workers. https://www.jfrog.com/confluence/display/RTF/Checksum-Based+Storage#Checksum-BasedStorage-ConfiguringtheMigrationProcess

tl;dr decrease artifactory.sha2.migration.job.queue.workers to somewhere around 2 - 2 * number of cores.

What you are experiencing is an exhaustion of the ThreadPoolExecutor.
The size of the thread pool by default is 4 * [number of cores].
In your case 64 (16 cores * 4)

However the limit on the number of threads you have configured for the sha256 migration job is 100.

On an artifactory instance without any load, this will not cause any failures, because there is a queue backing up the thread pull (in your case the size is 10000).

In your case the thread pull and the queue are full.

If I understand correctly you have overcame the issue. But for other people bumping into this thread, I would recommend decreasing the value of artifactory.sha2.migration.job.queue.workers to no more than half of the thread pull number of threads.

So in your case 32: (16 * 4)/2