Out of memory on fileupload

Discussion:

David Karlsen

2018-11-16 17:36:17 UTC

I'm struggling with outofmemory exception when using the following:

public Response upload( String token, File file ) {
FsDepot fsDepot = getApi();
Attachment attachment = new AttachmentBuilder()
.mediaType( "application/gzip" )
.contentDisposition( new ContentDisposition( "form-data;
name=\"data\" ; filename=\"" + file.getName() + "\"" ) )
.object( file )
.build();

Attachment tokenAttachment = new AttachmentBuilder()
.object( token )
.contentDisposition( new ContentDisposition( "form-data;
name=\"token\"" ) )
.mediaType( MediaType.TEXT_PLAIN )
.build();

MultipartBody multipartBody = new MultipartBody( Arrays.asList(
attachment, tokenAttachment ) );

return fsDepot.upload( multipartBody.getAllAttachments() );
}

protected <T> T getApi( Class<T> clazz, int timeoutInSeconds, URL
apiBaseAddress, boolean verbose ) {
List providers = Arrays.asList( new MultipartProvider(), new
FormEncodingProvider<>() );

LoggingFeature loggingFeature = new LoggingFeature(null, null,
64000, false, false);
T t = JAXRSClientFactory.create( apiBaseAddress.toExternalForm(),
clazz, providers, verbose ?
Collections.singletonList( loggingFeature ):
Collections.emptyList(),
null );
ClientConfiguration clientConfiguration = WebClient.getConfig( t );
clientConfiguration.getRequestContext().put(
Message.MAINTAIN_SESSION, Boolean.TRUE );

HTTPConduit httpConduit = clientConfiguration.getHttpConduit();
TLSClientParameters tlsClientParameters = new TLSClientParameters();
tlsClientParameters.setDisableCNCheck( true );
tlsClientParameters.setTrustManagers( new TrustManager[]{new
TrustAllTrustManager()});
httpConduit.setTlsClientParameters( tlsClientParameters );
HTTPClientPolicy httpClientPolicy = httpConduit.getClient();

httpClientPolicy.setAllowChunking( false );
httpClientPolicy.setConnection( ConnectionType.KEEP_ALIVE );
httpClientPolicy.setConnectionTimeout( TimeUnit.SECONDS.toMillis( 2 ) );
//have higher timeout on network than towards ADCM
httpClientPolicy.setReceiveTimeout( TimeUnit.SECONDS.toMillis(
timeoutInSeconds + 10 ) );
httpClientPolicy.setAutoRedirect( false );
httpConduit.setClient( httpClientPolicy );

return t;
}

--
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

Joe Schaefer

2018-11-16 17:59:50 UTC

Permalink

You probably need to use a stream instead of loading all of the attachments
into RAM prior to delivery.

Post by David Karlsen
public Response upload( String token, File file ) {
FsDepot fsDepot = getApi();
Attachment attachment = new AttachmentBuilder()
.mediaType( "application/gzip" )
.contentDisposition( new ContentDisposition( "form-data;
name=\"data\" ; filename=\"" + file.getName() + "\"" ) )
.object( file )
.build();
Attachment tokenAttachment = new AttachmentBuilder()
.object( token )
.contentDisposition( new ContentDisposition( "form-data;
name=\"token\"" ) )
.mediaType( MediaType.TEXT_PLAIN )
.build();
MultipartBody multipartBody = new MultipartBody( Arrays.asList(
attachment, tokenAttachment ) );
return fsDepot.upload( multipartBody.getAllAttachments() );
}
protected <T> T getApi( Class<T> clazz, int timeoutInSeconds, URL
apiBaseAddress, boolean verbose ) {
List providers = Arrays.asList( new MultipartProvider(), new
FormEncodingProvider<>() );
LoggingFeature loggingFeature = new LoggingFeature(null, null,
64000, false, false);
T t = JAXRSClientFactory.create( apiBaseAddress.toExternalForm(),
clazz, providers, verbose ?
Collections.emptyList(),
null );
ClientConfiguration clientConfiguration = WebClient.getConfig( t );
clientConfiguration.getRequestContext().put(
Message.MAINTAIN_SESSION, Boolean.TRUE );
HTTPConduit httpConduit = clientConfiguration.getHttpConduit();
TLSClientParameters tlsClientParameters = new TLSClientParameters();
tlsClientParameters.setDisableCNCheck( true );
tlsClientParameters.setTrustManagers( new TrustManager[]{new
TrustAllTrustManager()});
httpConduit.setTlsClientParameters( tlsClientParameters );
HTTPClientPolicy httpClientPolicy = httpConduit.getClient();
httpClientPolicy.setAllowChunking( false );
httpClientPolicy.setConnection( ConnectionType.KEEP_ALIVE );
httpClientPolicy.setConnectionTimeout( TimeUnit.SECONDS.toMillis( 2 ) );
//have higher timeout on network than towards ADCM
httpClientPolicy.setReceiveTimeout( TimeUnit.SECONDS.toMillis(
timeoutInSeconds + 10 ) );
httpClientPolicy.setAutoRedirect( false );
httpConduit.setClient( httpClientPolicy );
return t;
}
--
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

David Karlsen

2018-11-16 18:12:44 UTC

Permalink

.object( new FileInputStream(file)) ?

Post by Joe Schaefer
You probably need to use a stream instead of loading all of the
attachments into RAM prior to delivery.

--
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

Joe Schaefer

2018-11-17 11:33:43 UTC

Permalink

Yes, something like that. I'm not a Java programmer, but I'm sure there's
a way to do file uploads using streams and callbacks instead of copying the
full file contents into RAM before submitting.

Post by David Karlsen
.object( new FileInputStream(file)) ?

Post by Joe Schaefer
You probably need to use a stream instead of loading all of the
attachments into RAM prior to delivery.