[TEST] [LTS-2023] missing directories when testing with local contribution

[cheshire]

hello, i need to test operations and listeners using vocabularies, they are present in my studio package but, from what i understand are not included when the package is deployed locally  upon testing

• this error when running a test meant to use a local directory contribution

%ERROR  1,should_Pass(my.package.operations.SomeOperationTest)
%TRACES 
org.nuxeo.ecm.directory.DirectoryException: Failed to invoke operation SomeOperation, No directory registered with name: Some_Directory

• below is the structure of my core project source folder

main
  |- java
  |   |- my.package.operations
  |   |   |- SomeOperation.java
test
  |- java
  |   |- my.package.operations
  |   |   |- SomeOperationTest.java
  |- resources
  |   |- directories
  |   |   |- Some_Directory.csv
  |   |- OSGI-INF
  |   |   |- test-directories-contrib.xml

• here is my test class

@RunWith(FeaturesRunner.class)
@Features(AutomationFeature.class)
@RepositoryConfig(cleanup = Granularity.METHOD)
@Deploy({
    "studio.extensions.my-studio",
    "my-bundle-symbolic-name",
    "my-bundle-symbolic-name:OSGI-INF/test-directories-contrib.xml"})
public class SomeOperationTest {

  @Inject
  OperationContext context;

  @Inject
  AutomationService automation;
  
  public void should_Pass() throws OperationException {

    automation.run(context, "SomeOperation"); // raises exception

    // ... some assertion
  }
}

• here is the operation failing

@Operation(
    id = "SomeOperation",
    category = "Some",
    label = "Operation",
    description = "Openning a directory")
public class SomeOperation {

  @OperationMethod()
  public void run() {

    Framework.getService(DirectoryService.class).open("Some_Directory"); // raises exception

    // ... some more code
  }
}

• this is the contribution

<?xml version="1.0"?>
<component name="my.package.test.directories">
  <extension target="org.nuxeo.ecm.directory.GenericDirectory" point="directories">
    <directory name="Some_Directory" extends="template-directory">
      <schema>vocabulary</schema>
      <idField>id</idField>
      <autoincrementIdField>false</autoincrementIdField>
      <dataFile>directories/Some_Directory.csv</dataFile>
      <createTablePolicy>on_missing_columns</createTablePolicy>
    </directory>
  </extension>
</component>

• this is the content of the csv file

id,label,obsolete
"some","entry","0"

what am i missing, i have been running in circle for quite some time now i can't seem to find a solution anywhere

[Jordann_Rousse1]

Can you try to add the DirectoryFeature in your test class?

Instead of:

@Features(AutomationFeature.class)

Try:

import org.nuxeo.directory.test.DirectoryFeature;
...
@Features({ AutomationFeature.class, DirectoryFeature.class })

And when running the tests, have a look in the console for the "Component loading" part, if for some reason something cannot be loaded, it will show up there (and tell you what dependencies is missing)

[cheshire]

i unfortunately get the same error.

[Jordann_Rousse1]

Do you have any components unable to be loaded during the test? In the console, have a look for the "Component Loading Status" part, you should have something like below that could maybe explain why your vocabulary is not loaded:

======================================================================
= Component Loading Status: Pending: 0 / Missing: 3 / Unstarted: 1 / Total: 206
  * service:org.nuxeo.elasticsearch.managment.contrib references missing [target=org.nuxeo.ecm.core.management.CoreManagementComponent;point=probes, target=org.nuxeo.ecm.core.management.CoreManagementComponent;point=healthCheck]
  * service:org.nuxeo.ecm.core.automation.features.bindings references missing [target=org.nuxeo.ecm.automation.server.AutomationServer;point=bindings]
  * service:org.nuxeo.runtime.stream.service.managment.contrib references missing [target=org.nuxeo.ecm.core.management.CoreManagementComponent;point=probes, target=org.nuxeo.ecm.core.management.CoreManagementComponent;point=healthCheck]
  - service:org.nuxeo.ecm.platform.ec.notification.service.NotificationService
======================================================================

[cheshire]

i do

======================================================================
= Component Loading Status: Pending: 6 / Missing: 3 / Unstarted: 0 / Total: 197
  * service:org.nuxeo.core.elasticsearch.custom requires [service:org.nuxeo.elasticsearch.ElasticSearchComponent.contrib]
  * service:org.nuxeo.ecm.platform.suggestbox.core.defaultSuggesters.override requires [service:org.nuxeo.ecm.platform.suggestbox.core.defaultSuggesters]
  * service:my.package.services.AuthService requires [service:org.nuxeo.ecm.jwt.auth, service:org.nuxeo.ecm.restapi.server.auth.config]
  * service:org.nuxeo.ecm.platform.thumbnail.listener.disable requires [service:org.nuxeo.ecm.platform.thumbnail.listener]
  * service:my.package.core.marshaller requires [service:org.nuxeo.ecm.platform.types.marshallers]
  * service:org.nuxeo.ecm.document.pageproviders.override requires [service:org.nuxeo.ecm.document.pageproviders]
  * service:studio.extensions.autodistribution references missing [target=org.nuxeo.elasticsearch.ElasticSearchComponent;point=elasticSearchIndex, target=org.nuxeo.ecm.platform.ui.web.auth.service.PluggableAuthenticationService;point=loginScreen, target=org.nuxeo.ecm.platform.WebResources;point=resources, target=org.nuxeo.ecm.platform.WebResources;point=bundles, target=org.nuxeo.ecm.platform.types.TypeService;point=types, target=org.nuxeo.ecm.platform.ec.notification.service.NotificationService;point=templates]
  * service:org.nuxeo.ecm.core.automation.features.bindings references missing [target=org.nuxeo.ecm.automation.server.AutomationServer;point=bindings]
  * service:org.nuxeo.runtime.stream.service.managment.contrib references missing [target=org.nuxeo.ecm.core.management.CoreManagementComponent;point=probes, target=org.nuxeo.ecm.core.management.CoreManagementComponent;point=healthCheck]
======================================================================

is there something i can do to load them, i tried @Deploy  but end up with deployment errors

[Jordann_Rousse1 · Accepted solution]

As you can see, your studio contribution is not loaded due to missing references, which should explain why your vocabulary is not defined during the context of your unit test:

  * service:studio.extensions.autodistribution references missing [target=org.nuxeo.elasticsearch.ElasticSearchComponent;point=elasticSearchIndex, target=org.nuxeo.ecm.platform.ui.web.auth.service.PluggableAuthenticationService;point=loginScreen, target=org.nuxeo.ecm.platform.WebResources;point=resources, target=org.nuxeo.ecm.platform.WebResources;point=bundles, target=org.nuxeo.ecm.platform.types.TypeService;point=types, target=org.nuxeo.ecm.platform.ec.notification.service.NotificationService;point=templates]

For each missing reference, you need to find which bundle brings it, and you should deploy it within your unit test.
For example with point=elasticSearchIndex, we can see the extension point is defined there: https://github.com/nuxeo/nuxeo/blob/1b35dcdbce963594a6d891e7ccdf0c9279ad4fce/modules/platform/nuxeo-... 

Which is part of bundle "org.nuxeo.elasticsearch.core" (see here https://github.com/nuxeo/nuxeo/blob/1b35dcdbce963594a6d891e7ccdf0c9279ad4fce/modules/platform/nuxeo-...

So either you deploy "org.nuxeo.elasticsearch.core", or you can directly use feature "RepositoryElasticSearchFeature" already loading that bundle:  https://github.com/nuxeo/nuxeo/blob/1b35dcdbce963594a6d891e7ccdf0c9279ad4fce/modules/platform/nuxeo-... 

Repeat that logic for each missing references until your service:studio.extensions.autodistribution is loaded when you execute your test! 🙂 

[cheshire]

@Jordann_Rousse1 i will say that i needed to copy some contributions from Github because test jars where missing

Offline / Missing artifact org.nuxeo.ecm.platform:nuxeo-platform-notification:jar:tests:2023.24.1Java(0)

if you happen to have a solution for this as well, that would be amazing, since what i did work based one what you told me, you are still the GOAT,  i no longer require directories contributions for tests so kudos indeed

[Jordann_Rousse1]

Hey can you share your pom.xml so I could try to reproduce please? Maybe you're missing a Maven repository, you should have most JARs there (+ nuxeo-studio repo for your Studio project):

        <repository>
            <id>maven-public</id>
            <url>https://packages.nuxeo.com/repository/maven-public/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <updatePolicy>always</updatePolicy>
                <enabled>true</enabled>
            </snapshots>
        </repository>

[cheshire]

the contribution copied from github at the end is what allows me to bypass the need for nuxeo-platform test-jar (which is commented out in the core project's pom)

• below is the parent pom

<?xml version="1.0"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://maven.apache.org/POM/4.0.0"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.nuxeo</groupId>
    <artifactId>nuxeo-parent</artifactId>
    <version>2023.24.1</version>
  </parent>

  <groupId>my.group.id</groupId>
  <artifactId>my-project-parent</artifactId>
  <version>${revision}</version>
  <name>My Project Parent</name>
  <description />
  <packaging>pom</packaging>
  <properties>
    <skccore.version>2023.24.1</skccore.version>
    <nuxeo.target.platform.name>lts</nuxeo.target.platform.name>
    <nuxeo.target.platform.version>2023.*</nuxeo.target.platform.version>
    <revision>6.1.1</revision>
    <lombok.version>1.18.30</lombok.version>
    <nuxeo.skip.enforcer>true</nuxeo.skip.enforcer>
  </properties>

  <modules>
    <module>my-project-api</module>
    <module>my-project-core</module>
    <module>my-project-package</module>
  </modules>
  <licenses />
  <mailingLists />
  <issueManagement />
  <ciManagement />
  <developers />
  <repositories>
    <repository>
      <id>nuxeo-studio</id>
      <url>https://connect.nuxeo.com/nuxeo/site/studio/maven</url>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <updatePolicy>always</updatePolicy>
        <enabled>true</enabled>
      </snapshots>
    </repository>
    <repository>
      <id>maven-public</id>
      <url>https://packages.nuxeo.com/repository/maven-public/</url>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <updatePolicy>always</updatePolicy>
        <enabled>true</enabled>
      </snapshots>
    </repository>
    <repository>
      <id>maven-private</id>
      <url>https://packages.nuxeo.com/repository/maven-private/</url>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <updatePolicy>always</updatePolicy>
        <enabled>true</enabled>
      </snapshots>
    </repository>
  </repositories>
  <pluginRepositories>
    <pluginRepository>
      <id>maven-public</id>
      <url>https://packages.nuxeo.com/repository/maven-public/</url>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </pluginRepository>
  </pluginRepositories>
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>nuxeo-studio</groupId>
        <artifactId>my-studio-project</artifactId>
        <version>${revision}</version>
        <scope>test</scope>
      </dependency>
      <dependency>
        <groupId>org.nuxeo</groupId>
        <artifactId>nuxeo-parent</artifactId>
        <scope>import</scope>
        <type>pom</type>
        <version>${nuxeo.platform.version}</version>
      </dependency>
      <dependency>
        <groupId>my.group.id</groupId>
        <artifactId>my-project-api</artifactId>
        <version>${project.version}</version>
      </dependency>
      <dependency>
        <groupId>my.group.id</groupId>
        <artifactId>my-project-package</artifactId>
        <version>${project.version}</version>
      </dependency>
      <dependency>
        <groupId>my.group.id</groupId>
        <artifactId>my-project-core</artifactId>
        <version>${project.version}</version>
      </dependency>
      <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>${lombok.version}</version>
      </dependency>
      <dependency>
        <groupId>com.okta.jwt</groupId>
        <artifactId>okta-jwt-verifier</artifactId>
        <version>0.5.7</version>
      </dependency>
      <dependency>
        <groupId>com.okta.jwt</groupId>
        <artifactId>okta-jwt-verifier-impl</artifactId>
        <version>0.5.7</version>
      </dependency>
      <dependency>
        <groupId>org.hamcrest</groupId>
        <artifactId>hamcrest-all</artifactId>
        <version>1.3</version>
        <scope>test</scope>
      </dependency>
      <dependency>
        <groupId>io.rest-assured</groupId>
        <artifactId>rest-assured</artifactId>
        <version>5.5.2</version>
        <scope>test</scope>
      </dependency>
      <dependency>
        <groupId>commons-validator</groupId>
        <artifactId>commons-validator</artifactId>
        <version>1.7</version>
      </dependency>
      <dependency>
        <groupId>net.javacrumbs.json-unit</groupId>
        <artifactId>json-unit-assertj</artifactId>
        <version>3.5.0</version>
        <scope>test</scope>
      </dependency>
      <dependency>
        <groupId>com.rabbitmq</groupId>
        <artifactId>amqp-client</artifactId>
        <version>5.22.0</version>
      </dependency>
      <dependency>
        <groupId>io.github.vaa25</groupId>
        <artifactId>poiji2</artifactId>
        <version>1.5.0</version>
      </dependency>
    </dependencies>
  </dependencyManagement>
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-jar-plugin</artifactId>
          <version>3.2.0</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</project>

• here is the pom of the core project i am currently testing

<?xml version="1.0"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>my.group.id</groupId>
    <artifactId>my-project-parent</artifactId>
    <version>${revision}</version>
  </parent>
  <artifactId>my-project-core</artifactId>
  <name>My Project Core</name>
  <description />
  <properties>
    <nuxeo.skip.enforcer>true</nuxeo.skip.enforcer>
    <maven.deploy.skip>true</maven.deploy.skip>
  </properties>
  <dependencies>
    <dependency>
      <groupId>nuxeo-studio</groupId>
      <artifactId>my-studio-project</artifactId>
      <scope>test</scope>
    </dependency>
    <!-- nuxeo -->
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-types</artifactId>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-publisher</artifactId>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-usermanager</artifactId>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-content-template-manager</artifactId>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.automation</groupId>
      <artifactId>nuxeo-automation-features</artifactId>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-thumbnail</artifactId>
    </dependency>

    <!-- managed -->
    <dependency>
      <groupId>org.apache.poi</groupId>
      <artifactId>poi</artifactId>
    </dependency>
    <dependency>
      <groupId>org.apache.poi</groupId>
      <artifactId>poi-ooxml</artifactId>
    </dependency>
    <dependency>
      <groupId>com.okta.jwt</groupId>
      <artifactId>okta-jwt-verifier</artifactId>
    </dependency>
    <dependency>
      <groupId>com.okta.jwt</groupId>
      <artifactId>okta-jwt-verifier-impl</artifactId>
      <scope>runtime</scope>
    </dependency>
    <dependency>
      <groupId>commons-validator</groupId>
      <artifactId>commons-validator</artifactId>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.elasticsearch</groupId>
      <artifactId>nuxeo-elasticsearch-core</artifactId>
    </dependency>
    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>io.github.vaa25</groupId>
      <artifactId>poiji2</artifactId>
    </dependency>

    <!-- local -->
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-pool2</artifactId>
      <!-- including version fails all tests -->
      <!-- <version>2.7.0</version>  -->
    </dependency>

    <!-- test -->
    <!-- ES test feature (provides org.nuxeo.elasticsearch.test.ElasticsearchFeature) -->
    <dependency>
      <groupId>org.nuxeo.elasticsearch</groupId>
      <artifactId>nuxeo-elasticsearch-core</artifactId>
      <type>test-jar</type>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-mail</artifactId>
      <type>test-jar</type>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-convert</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-rendition-core</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-io-core</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-default-config</artifactId>
      <scope>test</scope>
    </dependency>
    <!-- FIXME: allow the use of the NotificationFeature, currently not public ?
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-notification</artifactId>
      <type>test-jar</type>
      <scope>test</scope>
    </dependency>
    -->
    <dependency>
      <groupId>org.nuxeo.ecm.automation</groupId>
      <artifactId>nuxeo-automation-test</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-web-resources-core</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-login-jwt</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-platform-suggestbox-core</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.nuxeo.ecm.platform</groupId>
      <artifactId>nuxeo-rest-api-test</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.hamcrest</groupId>
      <artifactId>hamcrest-all</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>net.javacrumbs.json-unit</groupId>
      <artifactId>json-unit-assertj</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.awaitility</groupId>
      <artifactId>awaitility</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

• to remove missing components i am deploying this

@RunWith(FeaturesRunner.class)
@Features({
    MockitoFeature.class,
    RestServerFeature.class,
    StudioFeature.class,
    RuntimeStreamFeature.class,
    RepositoryElasticSearchFeature.class })
@Deploy("org.nuxeo.ecm.jwt")
@Deploy("org.nuxeo.ecm.core.management")
@Deploy("org.nuxeo.ecm.platform.suggestbox.core")
@Deploy("org.nuxeo.ecm.platform.thumbnail")
@Deploy("org.nuxeo.ecm.platform.convert")
@Deploy("org.nuxeo.ecm.platform.rendition.core")
@Deploy("org.nuxeo.ecm.platform.io.core")
@Deploy("org.nuxeo.ecm.default.config")
@Deploy("my-core-bundle-symbolic-name")
@Deploy("studio.extensions.my-studio-project")

• StudioFeature looks like this

@Deploy("org.nuxeo.ecm.platform.notification")
@Deploy("my-core-test-bundle-symbolic-name:OSGI-INF/default-general-settings-contrib.xml")
@Deploy("org.nuxeo.web.resources.core")
public class StudioFeature implements RunnerFeature {}

•  the contribution i got from notification-service-contrib looks like this

<?xml version="1.0"?>

<component name="org.nuxeo.notifications.test.config">

  <extension target="org.nuxeo.ecm.platform.ec.notification.service.NotificationService" point="generalSettings">
    <settings>
      <serverPrefix>http://localhost:8080/nuxeo/</serverPrefix>
      <eMailSubjectPrefix>[Nuxeo]</eMailSubjectPrefix>
      <mailSenderName>default</mailSenderName>
    </settings>
  </extension>

</component>