Tests failing on CentOS 7 build

1

Hello,

I’m building TileDB on a CentOS 7 VM with S3 compatibility. I’m consistently running into an issue where my code throws

*** Error in `./S3test’: free(): invalid pointer: 0x00007f917f7b5140 ***

this is also true when I run “make check” where I get

*** Error in `/home/cleader/TileDB/build/tiledb/test/tiledb_unit’: free(): invalid pointer: 0x00007ff1116c3120 ***

and

1: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1: tiledb_unit is a Catch v2.2.1 host application.

1: Run with -? for options

1:

1: -------------------------------------------------------------------------------

1: Backwards compatibility: Test error opening 1.3.0 array

1: -------------------------------------------------------------------------------

1: /home/cleader/TileDB/test/src/unit-backwards_compat.cc:79

1: …

1:

1: /home/cleader/TileDB/test/src/unit-backwards_compat.cc:79: FAILED:

1: due to a fatal error condition:

1: SIGABRT - Abort (abnormal termination) signal

1:

1: 0.000 s: Backwards compatibility: Test error opening 1.3.0 array

1/1 Test #1: tiledb_unit …***Exception: SegFault 0.03 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.03 sec

The following tests FAILED:

1 - tiledb_unit (SEGFAULT)

Errors while running CTest

I tried compiling without AVX2 (although my systems for sure supports it) and I get the same issue.

I’m building the latest dev version. I can provide more details about the backtrace if necessary.

Many thanks

2

Hi @cleader,

A backtrace would be very helpful, as well as the build options you used. Note that the AWS credentials need to be exported (or saved in $HOME via aws login if you have the CLI tools), although lack of credentials should not cause a crash like this.

Thanks,
Isaiah

3

Also note that for local testing of an S3 build, you need to do the following before running tiledb_unit or ctest (this starts minio in the background, assuming you have docker installed):

./scripts/run-minio.sh
export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=miniosecretkey
4

Hi,

My build options were “…/bootstrap --disable-avx2 --disable-werror --enable-s3” followed by make. I installed the aws-sdk-cpp separately with only the S3 options selected. Using cmake 3.14.6 and the standard zlib-devel, openssl-devel, curl-devel and bzip2 that come with CentOS 7 update manager.

I get the same error in the examples/cmake_project test too, and there no AWS functionality is used. If the library is built with AWS enabled do you always need to ensure these access keys are correct even if no AWS functionality is being actively called?

For what it’s worth, I was able to build the code the same way on my mac and everything worked fine. I’m going to try out the Docker image for now.

Thanks,
Chris

5

No, credentials are only required for s3://... URLs.

All of that should be ok except the separate AWS SDK installation – the SDK iterates version numbers very quickly, and we’ve only tested against the current pinned version in cmake/Modules/FindAWSSDK_Ep.cmake (1.7.108).

That said, we frequently build/test on Amazon Linux, which is similar to CentOS 7, and we use bootstrap ... --force-build-all-deps in order to make self-contained binaries with known-good versions of all dependencies – I would suggest trying --force-build-all-deps.

If you can share a backtrace, it might give us more insight in to what is happening.

6

Hi Isaiah,

You nailed it again! Must have been something about the AWS shared object I had displayed on my LD_LIBRARY_PATH not playing nicely with the tiledb library. I force built all the dependencies and now my tests are building.

For what it’s worth, the --disable-werror flag for the cmake setup does not propagate to the dependencies, so I had to restart the AWS install after running a find-and-sed command to strip the -Werror flags from the build files in order for the build to complete.

thanks again!

2 Likes
7

That’s great, thanks for the update.

For what it’s worth, the --disable-werror flag for the cmake setup does not propagate to the dependencies

I opened Make sure --disable-werror is passed to dependencies when possible · Issue #1402 · TileDB-Inc/TileDB · GitHub about this.

8

Great!

I look forward to testing these libraries now these wrinkles are all ironed out. Bucket creation/write/read all working.

thanks again